Migration: It isn’t just for the Birds

2. Migration: It isn’t just for the Birds Presented by Becky Bell MPLA/NDLA/SDLA Tri-Conference October 5, 2002

3. Data Migration • What is it? • Who does it? • How can you prepare? • When is it done? • Data Mapping • MARC Records • Non-MARC Records

4. Migration Considerations • Data Mapping • Serials Migration Components • Data Mapping Detail • MARC records • Non-MARC records • Data Cleanup

5. What is data migration? • Simply, moving data existing in one software application to a different software application • Usually is more complex than expected due to • Changed functionality • Field size incompatibility • Disparate record structures • Existing/new/no standards supported in either previous or successor software

6. Two “Truths” about Data Migration • 1. There is always a bigger mess than you thought. • 2. Your assumptions are always wrong. • From “An ABC guide to data migration,” by Ross Bentley, Computer Weekly, Feb 7, 2002, p.39, and attributed to Richard John, managing director of data migration at Alchemy. • “Data migration not the most fascinating aspect of implementing a new system.”

7. Data migration is a matter of ABC • Analysis of data • Building conversion programs • Cleaning incorrect data • Remember GIGO--Garbage in, garbage out

8. Who does data migration? • Any entity (library, library consortium, banks, retail stores, etc.) moving data from one software application to a different application • Focus today is library data • Ideally, library staff and vendor work together to create data mapping specifications

9. How can you prepare? • Read and learn about • Windows • MARC21 Holdings Format • Know what you currently have • Become familiar with your system’s data dictionary • Document your workflow • Document local practices • Work on data cleanup • Learn as much about your new system as possible • Talk to others who have completed their migration

10. When is it done? • Obviously, the switch to production • Milestones along the way • Decisions made about what must migrate • Data mapping specifications written • Subset(s) converted, loaded, and indexed • Data reviewed for mapping and data accuracy • As many iterations as needed to get what you expect

11. Keep in mind that if data migration is skimped, muffed or even rushed, the success of the new system can be dangerously compromised. • Details matter!

12. Migration Considerations • What data must be migrated? • What data are you willing to re-key? • What data are you willing to lose? • What data are you willing to share if you’re part of a consortium? • Can uncataloged/unlinked items exist in your new system?

13. Migration Considerations • Can bibliographic records exist without items and/or holdings records? • How much training and/or consulting services from the vendor can you afford? • Pre-conversion training? • Systems administration? • Functional training? • Remember that policies and privileges aren’t migrated; you’ll need to create them in new system.

14. Serials Migration Components • Standards • MARC21 Bibliographic Format • MARC21 Authority Format • MARC21 Holdings Format • NISO Z39.71-1999 Holdings Statements for Bibliographic Items • No standards for • Subscription, claiming, routing data • Acquisitions data (Orders, Vendors, Budgets)

15. Serials Migration Components • Serials • Serial data will be stored in bibliographic, holdings, item, acquisitions, and serials modules • Won’t look at Patrons, ILL, Booking mapping today • Linkages to maintain or create: serial record to bibliographic and holdings records to items and orders • Serials records not used for check-in can be mapped to a bibliographic record (and can be suppressed from the public catalog)

16. Data Mapping • Two types of library records • MARC • Bibliographic • Authority • Holdings • Non-MARC • Item

17. Data mapping • Patrons • Loans • Holds • Cash Transactions • Blocks • Vendors • Orders • Monograph • Serials • Standing orders • Purchase orders (invoices)

18. Data mapping • Budgets • Serials • Subscription records • Publication patterns • Claims • Routing lists • Binding records • Interlibrary Loan records • Booking records

19. Data mapping detail • MARC bibliographic records • If consortium, need to choose system architecture first (if there are options) • One bibliographic/authority/holdings database for all or separate databases for each library? • If one bibliographic database, migrate each library’s record or merge like records? • If records merged, what are the merge criteria? • Physical union catalog or virtual union catalog?

20. Data Mapping Detail • Examples of differences in bibliographic records for Cricket and Library Journal • Cricket cataloged by Jamestown College (NDJ) on January 24, 1991 and by Fargo Public Library (NFG) on February 1, 2000 • Differences • NFG has a revision date in 010 • NFG has additional subfields in 040 ($$d NST $$d NSD $$d DLC $$d IUL) • Tags 260, 300, and 321 formatted slightly differently, probably based on cataloging rules current when cataloged

21. Data Mapping Detail • 362 tag formatted pre-Z39.71-1999 Holdings Statements for Bibliographic Items in NDJ and post-standard in NFG • 901 tag capitalization differs • 936 tag identifies different surrogates used for cataloging • Additions • NDJ record has 265 and 350 tags included • NDJ added local holdings information in 500 tag • NFG record has 037 and 130 tags included • NFG record has two 500 tag notes

22. Data Mapping Detail • NFG added a 590 tag for local holdings information • NFG includes 850, 890, and 994 tags

23. Data Mapping Detail • Library Journalcataloged by Augustana College (SDA) on May 2, 1991 and by Dakota Wesleyan University (SDW) on November 5, 1992 • Differences • In 012, SDA has $$i 8510 and $$m l • In 012, SDW has no $$I and has a $$m c • In 040, SDA has many more $$d’s than SDW • Contents of 265 different—which is more current? • In 321, SDA includes a $$b • Contents of 350 different—which is more current? • Second 650 in SDA record has a $$w

24. Data Mapping Detail • 780 in SDA record has a $$w • Contents of 936 are different in SDA and SDW • Additions • SDA record added a 310, a 362, nine 510 fields • What happens when the newest record overlays the first record loaded?

25. Data Mapping Detail • Possible merge criteria • Each record converted to new system • Pro--each library retains its cataloging and retains local cataloging policies • Con--lots of duplicates • First bibliographic record into system becomes record for all, based on one common field (bib id number) • Pro--cataloging may not be acceptable to all • Con--Keeps database size smaller

26. Data Mapping Detail • First bibliographic record is supplemented as additional records are converted • Pro--Additional access points get added for retrieval • Con--Records can become very long • Con--False search results become more likely for your patrons • Con--Improper MARC tags add information more appropriately stored in the holdings record

27. Data Mapping Detail • Merging algorithm determined, for example, 100, 245 subfield a, 250, 260 subfield b and c match exactly • Pro--De-duplication of database accomplished • Pro--All like titles merged into one record • Con--Like titles may be different formats • Sample specifications from PALS to Aleph mapping • No one right way--study options, then choose what provides results you desire

28. Data Mapping Detail • MARC Holdings records • Legacy library systems won’t have holdings records to convert, so records will be created • Specifications need to exist that detail what data goes into tags and subfields • NUC symbol, location, call number, and pre- and post-input stamp information • Serials captions • Enumeration and chronology • URL stored in 856

29. Data Mapping Detail • Local notes • Will need to create location/call number mapping document • Sample location map • Call number pick list • Call number (x) suppression • 099 call numbers or other non-standard numbers • SuDocs numbers • Sample specifications

30. Data Mapping Detail • Items • Uncataloged/unlinked items will need bibliographic records • Specifications need to include mapping bibliographic data in items to proper MARC fields for brief records • Links between subscription records and items need to be defined

31. Data Mapping Detail • Sample items • Sample specifications • Sample data map

32. Data Mapping Detail • Course reserves • If serials titles or issues are on reserve, need to preserve reserve information and status • Sample specifications • Sample data map

33. Data Mapping Detail • Vendors • Used by both acquisitions and serials • Acquisitions uses for ordering and paying • Serials uses for claiming missing issues • Sample specifications • Sample data map

34. Data Mapping Detail • Orders • Orders need to exist for active serials (check-ins) • Sample specifications • Sample data map

35. Data Mapping Detail • Purchase orders (invoices) • Payment history • Sample specifications • Sample data map

36. Data Mapping Detail • Serials • Prediction patterns • Library Journal example • Cricket example • Coding detail • Seasons • Free text • Supplements or other additions to regular pattern • Memberships

37. Data Mapping Detail • Inactive serials records • Decision/internal/history/public records • Sample serial records • Sample specifications • Sample data map

38. Data Clean-up • Bibliographic/Holdings records • Identify commonly misspelled words in records in your catalog (use http://www.aallnet.org/sis/tssis/tsll/22-03/obs-locl.htm as a source) and correct • Identify records with duplicate call numbers • Identify records with no call numbers • Browse your title index for incorrect filing characters (A, An, The) and correct • Begin discussions about indexing

39. Data Clean-up • Non-MARC records • Review reports for obvious errors • Weed collections • Resolve outstanding overdues, fines and bills • Inventory • Review existing notes and remove those no longer needed • Monitor binding lists

40. Data Clean-up • Purge closed orders and purchase orders (invoices) • Review open orders

41. Recommendations • Staff development issues • Windows • MARC21 Holdings Format training • Update (or create) procedure manuals to reflect current practices • Document local practices • Attend conference programs about standards and migration issues • Subscribe to open mailing lists for next vendor

42. Recommendations • Involve all staff in data mapping planning and data review • Develop a good sense of humor--it will be one of your most valuable tools • Remember you have a life outside of data migration!

43. In Closing, • Review your data • Write detailed conversion specifications • Review converted data by comparing data in old and new systems • Details matter!