COL: data acquisition, storage and preservation

1. COL: data acquisition, storage and preservation Roger Brugger.brugge@reading.ac.uk www.met.reading.ac.uk/~brugge Climatological Observers Link Department of MeteorologyUniversity of Reading Transmission, presentation and archiving of meteorological data, 5 Oct 2011

2. Outline • What is COL? • What the data are used for • Data gathering • Daily versus monthly data • Old and new methods • Data quality – station gradings • Data storage • What is stored? • How is it stored? • Data preservation • For use by COL • For use by others

3. What is COL? • June 1950 - An advert in Weather in June 1950 by Tom Suttie • enquiring about setting up an organisation for the exchange of weather data by amateurs • 1969 - A chance sighting of this advert by Eric Bonsor • there followed an exchange of letters, and • January 1970 - first edition of the Directory of Amateur Observers published • Site details, instrumentation and observing routines • June 1970 - first issue of the Bulletin • 48 stations reporting, describing the weather of May 1970. • Nowadays more than 330 sites each month • Aim of COL - to publish the monthly bulletin at the earliest opportunity • Achieved for over 41 years now • The COL Bulletin is now a rare source of comprehensive UK monthly weather summaries. • Membership details added to the Directory of Amateur Observers • Almost 1300 stations are now included, together with photographs of stations

4. Data gathering • Monthly data only • A few exceptions for a daily data table • COL cannot handle the volume of daily data that would otherwise accrue • Paper • The original method (15% of returns) • Email • Also used as a reminding mechanism to most members (15%) • Web (70% of returns) • Underpinned by a database • PHP5 MyAdmin and SQL >archiving • Provides members with an early snapshot • The way of the future?

5. Paper form – main data features only shown

6. Data gathering • Monthly data only • A few exceptions for a daily data table • The Bulletin • Paper • The original method (15% of returns) • Email • Also used as a reminding mechanism to most members (15%) • Web (70% of returns) • Underpinned by a database • PHP5 MyAdmin and SQL > archiving • Provides members with an early snapshot • The way of the future?

7. Email reply form – the start of a message for August 2011

8. Data gathering • Monthly data only • A few exceptions for a daily data table • The Bulletin • Paper • The original method (15% of returns) • Email • Also used as a reminding mechanism to most members (15%) • Web (70% of returns) • Underpinned by a database • PHP5 MyAdmin and SQL > archiving • Provides members with an early snapshot • The way of the future?

12. Data quality • COL welcomes observations from everyone • Multiple stations per town • Wide variety of instrumentation and exposure • Airports, universities, institutions and the hobbyist • Instrumentation • Originally based around Stevenson screen, mercury-in-glass thermometers and Snowdon raingauges • Many good quality sites • Increasing use being made of AWS equipment • Varying costs, and also performance • Need a way to distinguish between stations that (even in the same locality) might report widely differing obs • > COL station grading system (Stephen Burt)

13. Buxton, Norfolk Stratfield Mortimer Elderslie

14. Tivington Red Wharf Bay, Anglesey Compton Martin

15. Data quality • COL welcomes observations from everyone • Multiple stations per town • Wide variety of instrumentation and exposure • Airports, universities, institutions and the hobbyist • Instrumentation • Originally based around Stevenson screen, mercury-in-glass thermometers and Snowdon raingauges • Increasing use being made of AWS equipment • Varying costs, and also performance • Need a way to distinguish between stations that (even in the same locality) might report widely differing obs • > COL station grading system (Stephen Burt)

16. Station grading system(as used in 2011) Grading system development over the years has been led by Stephen Burt

18. Incl. SYNOP and CLIMAT sites

19. Station grade + photograph + station details = a reasonable idea of data quality

20. Data storage • What is stored? • Monthly summaries • A subset of the data; coverage 1940s onwards • Column-aligned text files • Easily manipulated by Excel and Fortran, etc. • Used to make quick calculations and climatological values • e.g. recently published 1981-2010 averages • All monthly data, numbers and text • Database; coverage 2006 onwards • Generated by SQL/webpage • Little-used • All back issues of the bulletin available electronically • 40th anniversary CD

21. Members observations – where do/could they go? • Lots of high-quality COL observations (especially grade A stations) • Effectively unused or un-archived simply because they are not reporting to UKMO • Data from these sites are likely to be lost unless special arrangements are made. Currently paper copies • traditionally, • sometimes saved in local history libraries • donated to an observer living close by • sent to the local rubbish tip (the majority) • now: they can be deposited with the Chiltern Observatory Trust • Electronic data • Monthly – some/all retained by COL • Daily – ‘delete’ key • Either way – file formats soon become outdated • A proper remote archiving/backup facility is preferable. • Might BADC have a role here? • Two suggested solutions…

22. Preserving members observations – Possible solution (1) • 2011: Many amateur sites are fully/partial AWS • Might consumer-AWS instrument manufacturers agree a common ‘archive file’ export standard with BADC? • to simplify the archiving process • ensure amateur data is not lost when a disk crashes or the observer dies. • If this was a file that could be exported then uploaded to a BADC site, say annually, it would take only minutes. • A mandatory metadata section should be required. • Might Davis… • who are the market leaders • who currently provide an export file format for NOAA • …take a lead on this?

23. Preserving members observations – Possible solution (2) • BADC to propose a standard Excel spreadsheet-type template for amateur observers to use. • This would greatly simplify the transmission and storage of amateur observations to BADC. • A mandatory metadata section should be required, perhaps including site plan/s and photographs from cardinal points. • Perhaps pilot • using the existing ‘COL archive’ template materials (hourly, daily, monthly templates have been created) • and with a voluntary participation of a handful of A-grade COL stations in 2012? • Issues: • There are as many observer-owned spreadsheet formats as observers • However, given a lead by BADC, COL members might be persuaded to modify or convert their formats

24. More information • This presentation – • To be placed at www.met.reading.ac.uk/~brugge • Website • www.met.reading.ac.uk/~brugge/col.html • Weather article • Brugge, R., 2010. Forty years of the Climatological Observers Link. Weather, 65, 139-143. • Email the author • r.brugge@reading.ac.uk • Bulletin – some copies available today • ‘Processed data’: an example • Burt, S.D. and R. Brugge, 2011. Climatological Averages for 1981-2010 and 2001-2010 for stations appearing in the monthly bulletin of the Climatological Observers Link. ISBN 9780956948502.