Data management

1. Data management LingDy February 13, 2012 TUFS, Tokyo David Nathan Endangered Languages Archive Hans Rausing Endangered Languages Project SOAS, University of London

2. Two most valuable strategies • design and use a filename system • work out your basic units of documentation and the relationships between them - if you get these right, it will do the “heavy lifting” of your data management strategy - data and metadata are intertwined, points in a spectrum rather than different things

3. Three most important qualities • consistency • machine readability “computer programs can act on data in terms of its proper structures and categories” bad example • documentation of conventions, structures, methods

4. Data management • understand and model the data (units, relationships) • use appropriate data structure methods – in both file contents and organisation • use appropriate and conventional data encoding methods (e.g. Unicode) • be explicit and consistent • plan for flow of data, working with others, across different systems • document steps, decisions, conventions, structures • think ahead to archiving

5. Managing data in your computer • design a well-organised system of folders so that you can always find your stuff according to what it is, not: • where the software decided to put it • what the software decided to call it • when/where you last used it • what someone else called it

6. File structures and names • design folder structure as a logical hierarchy that suits your goals, content and work style • have documentary materials within one overall directory (e.g. for backup) • make directories for relevant categories, e.g. sessions, media types, dates • design it so that you will always be able to find things • you may need to restructure at different points in your project, e.g. move from date-based to session-based structures

7. Designing a file/folder structure • it should relate to reality • locations should make sense, so you (and others) will know where to look for things (where do you keep your passport; favourite cup?) • the best location is “the place that one would naturally look to find it”

8. 3 models for file system structures • tree of descriptive folder- and file-names • one folder with descriptive filenames • one folder with numerical filenames … what else is needed?

9. On identifiers • real world objects are uniquely identified because they are physically unique - an unlabelled cassette is poorly identified • digital objects have no physical existence - they depend on identifiers that we give them • three types of identifiers: • semantic • keys • relative

10. On identifiers • semantic, e.g. • Nelson Mandela • The Sound of Music • SA_JA_Bongo_Palace_Land Dispute Trial_015_29-04-2010.wav * * SA_JA_Bongo_Palace_Land Dispute Trial_015_29-04-2010.wav

11. On identifiers • keys (disambiguators), e.g. • 1137204 (a student number) • 0803 211 6148 (a telephone number), p12893fh23.pdf (some system's reference number)

12. On identifiers • relative, e.g. • 67 High Street • the secretary • index.html • metadata.xls

13. On identifiers • your collection may have a mix of these but it is important to be aware of their differences and limitations, for example: • semantic identifiers: invite name clashes • keys: a program or process might depend on the identifier to work properly • relative identifiers: if you move them, you probably change or destroy their meaning

14. Objects and identities • a digital object’s identity includes its location • a file’s full identity = path + filename • the path is a representation of the volume and the directory (folder) hierarchy • if the full identity is unambiguous then everything can be fine, compare: • c:\\dogs\spaniels\rover.jpg • c:\\cars\british\rover.jpg or • lectures\syntax\2013-02-12\notes.doc

15. Objects and identities • but semantic identifiers are potentially dangerous, because just adding more chunks to disambiguate them will not work: • my\rover.jpg • my\white_rover.jpg • so domains that do not offer semantic uniqueness may need identifiers which are either keys, or relative identifiers

16. Segue to file names • (having said all that) • filenames are only filenames, and do not necessarily provide information • common mistaken assumptions: • that a filename “dp_verbs_39.wav” means there is an entity “dp_verbs_39” • that files are logically linked just by sharing some part of their filenames - these are only true if your system ensures it (and you state it explicitly)

17. File naming • filenames that are unsystematic or are non-standard will cause problems, eventually • unsystematic file naming might be OK if • you already have many files • you have a working method that already does everything you need to do • your “system” will do everything you need to do in the future

18. Manage file names from the start • a new file: • don’t just accept the default filename or location suggested by the application when you first save the file • put it where it belongs, immediately. If necessary, create the place (directory/path) where it belongs • name it according to your naming system! • if you have an inventory/index of files, add an entry for the new file

19. Filename rules • all filenames should have correct extensions • each filename should have only one ".", before the extension • use only ASCII characters (US keyboard) • use only letters, numbers, hyphens (-) and underscores (_) • keep filenames short, just long enough to contain the necessary identifier - don't fill them up with lots of information about the content (that is metadata!)

20. How about these file names? • ready.audio.wav • ReAlLyOdDtOReAd.txt • éclair.jpg • éclair_fr.jpg • e'clair.jpg • french-cake.jpeg • french-cake.jaypeg • lexicon-master • ɘɫIɲʰ.eaf • ice cream.doc • OBAMA.TXT • Obama.txt

21. Make filenames sortable • make filenames usefully sortable: • 20100119lecture.doc • 20100203lecture.doc

22. Associating files • you can make resources sortable together by giving them the same filename root (the part before the extension), or part of the root: • document your conventions and system if you do this

23. Avoid metadata in filenames • avoid putting metadata into filenames. A filename is an identifier, not a data container • better to use a simple (semantic) filename or a key (i.e. meaningless) filename, and then create a metadata table to contain all the relevant information • a table can properly express all the information, contain links etc, and is extensible for further metadata

24. Avoid metadata in filenames • e.g. Paaka_Reefs_Dan_BH_3Oct97.wav • better: • paaka_063.wav plus • paaka_063.txt paaka_063.txt

25. A filenaming system • carefully design a filename system for your data and document the system so that somebody else can understand it • one documenter’s new system: aaa_bb_cc_yyyy-mm-dd_nnn.wav

26. A filenaming system • aaa_bb_cc_yyyy-mm-dd_nnn.wav aaa = village code bb = (main) speaker code cc = genre/event code yyyy-mm-dd = date (why this order?) nnn = optional number (e.g. 001) .wav = correct extension for file content type

27. Also (for Part 2) • creating an inventory/index/metadata • metadocumentation • data/file versions • transferring data • sharing data • backup

28. Documenting the filename system • describe the system - how would you describe it? - where would you put the description? • document the codes – this is probably part of your metadata

29. On changing file names • decide if it’s possible, benefits and side effects (e.g. loss of links in ELAN files) • design a system first • don’t change names in situ – copy data set and gradually migrate it to your new system • document file name changes • if possible, automate or copy and paste filenames • if possible, use machine processes, e.g. system filename listings, XLS formulas