1 / 49

RAP Champions Network Meet Up #3

RAP Champions Network Meet Up #3. 10 th October 2019. Menu. 11.00 Welcome and recap - Martin Ralphs (15 mins) 11.30 Revisiting the RAP levels - Alexander Newton and Joshua Halls (45 mins) 12.00 QA of models – the DfE experience – Nicola Brassington (30 mins)

floyd
Télécharger la présentation

RAP Champions Network Meet Up #3

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. RAP Champions Network Meet Up #3 10th October 2019

  2. Menu 11.00 Welcome and recap - Martin Ralphs (15 mins) 11.30 Revisiting the RAP levels - Alexander Newton and Joshua Halls (45 mins) 12.00 QA of models – the DfE experience – Nicola Brassington (30 mins) 12.30 Lunch (provided) and networking 13.15 Survey results: data science QA practices (Martin Ralphs) (15 mins) 13.30 QA of coding – developing guidance – Joshua Halls (30 mins) 14.00 Discussion: where next for the RAP champions network? (workshop session: 90 mins) 15.30 Wrap up

  3. Last week…

  4. WE WON THE GSS Collaboration Award! Photo by Nicolas Tissot on Unsplash

  5. NHS NSS were the runner up in the GSS Methods award! Photo by Nicolas Tissot on Unsplash

  6. Be sure to download Dr Dray’s limited edition GSS Awards RAP Sticker! You know you want it!

  7. We’ve been invited by the Statistical Journal of the International Association of Official Statistics to submit a paper on how we’re building reproducible analysis into UK government statistics • We need some volunteers from mature RAP project teams to work with us on this alongside ONS / GDS. You’ll get a co-author credit, of course. • Great opportunity to get into a peer-reviewed scientific journal

  8. What they said: “I think a contribution here would be very good internationally. I imagine there are similar initiatives on a smaller scale going on elsewhere, but the way you guys have set it up as a discipline and tied in the whole Data Science reproducibility idea into operations excellence is fantastic. If you can do something I think it would help many others as would provide a great reference and materials. The SJIAOS is a good vehicle for this - part of ISI and has a broad reach.”

  9. 2018 Barriers and blockersCulture and Organisation • “High priests” and “black boxes” – so RAP is hard to sustain • Complacency through too much automation? • Fears about job security • Lack of time for RAP projects

  10. 2018 Barriers and blockersData and Technical • “Only applicable if we use R and Python” • Data storage and access, especially for new projects • Lack of clarity on data sensitivity and handling • Access to tools varies widely across departments • Difficulty of sharing workflows across multiple teams • Building mature pipelines into IT business as usual

  11. What did we say champs would do? • Support and advise others who want to RAP • Peer review code across organisations • Share good practice across the network through open code and case studies • Allow others to shadow us And we’ve started to • Develop standards and guidance

  12. Recap: stuff we’ve set up Blogs – the original RAPpers at GDS GDS MOOC and RAP Manual govdatascience Slack channel #rap_collaboration Duncan G’s RAP Github repository and sites for individual projects List of current RAP projects (Google Sheets) RAP champions network repository on GSS website NHS Services Scotland / DAPCATS “levels of RAP” to benchmark the progress and robustness of RAP projects Exploratory work on “Must Have, Nice to Have, Gold Standard” for workflows, tools, skills and standards

  13. Rethinking RAP Levels Group Discussion

  14. What should RAP levels help to achieve?

  15. Confusing two dimensions Is this RAP? Why is testing this late? Makes it harder to sell RAP

  16. How can we improve this? • Simplify • Condense into two categories • Essential RAP • Enhanced RAP • Different methods form components of each

  17. Should we mandate that projects apply either functional or object-oriented programming?  • Should testing be split into unit/integration testing? • Is testing of functions essential RAP? • Should there be a least one integration test for essential? • Is documentation essential? • And how much should be the minimal amount?

  18. In groups, discuss: • What do you think of the current RAP levels? • Is this the right approach to improving them? • Are two categories enough? • Are the components of each category right? • What would you add/delete/move? We have included the RAP Minimum Viable Products slides from last session’s discussion.

  19. The DfE Quality Assurance Framework: Re-platforming Models Into R Model Improvement and Assurance Unit Nicky Brassington

  20. Outline of presentation

  21. Quality Assurance within DfE • DfE has nearly 800 analysts • All 4 professions and unbadged • Embedded out across the department • Spread across 8 locations • The Permanent Secretary requires assurance that modelling/analysis risks are being appropriately managed • QA review in 2017-18 demonstrated requirement for central guidance to set standards

  22. The DfE Quality Assurance Framework • The framework aims to ensure that QA is applied consistently across the department • Aqua book used as the foundation • Outputs should: • Be as reliable and accurate as possible • Address the initial purpose of the work • Have a clear audit trail, ownership and be fully approved • The framework was implemented April 2019 • A QA log is required for every model/analysis with 3+ days of resource

  23. The Five Pillars of the QA Framework A quick guide to getting started with the framework Governance & Documentation All analysis require a commissioner and lead analyst Documentation should explain how the model/analysis works, inputs & outputs, and highlight risks See activities list Verification Has the analysis/model been built right? At a minimum all code & spread sheets require peer-review See activities list Data & Assumptions The reliability of the key inputs to the model should be known and their quality and limitations recorded See activities list Structure & Clarity Data analysis require a clear logical structure, facilitating easier QA, maintenance and development Best practice coding and spread sheet design should be used to maximise clarity See activities list Validation Has the right analysis/model been built? Methodology should be scrutinised to ensure that the selected approach is appropriate Outputs require sense-checking to ensure that they are reasonable, in the expected range and the sensitivities are understood See activities list 1 2 3 4 5 28

  24. Embedding a Culture of QA Since January initiated a range of activities • QA training >400 analysts, policy and finance colleagues • Workshops for Senior Responsible Owners • Established QA Officers’ Network >30 analysts • Produced guidance and templates • Coding standards • How to QA code • How to re-platform models

  25. Approaches to Modelling in DfE • A wide variety of software packages • Excel models range from straight-forward calculations through to multi-tab complex models with macros • Challenge ensuring robust QA • Over the last year resource used to redesign spreadsheets as R-based models

  26. Cross-Government Guidance on Re-platforming models to R • Growing experience within DfE as models are ported into R • Engaging with DfE RAP champions • Collaborating with OGDs to create cross-government guidance

  27. Re-platforming models to R Guidance • Guide comprises two sections • Why (or why not) you should re-platform • Practical guides and tips Should you Re-platform? • What are the benefits of porting a model? • RAP principles - QA and speed • What prevents people porting models? • Circumstances where porting may not be a good idea • How to ensure buy-in with your stakeholder

  28. Considerations Before Re-platforming • Support network to ensure • Code is optimised • Methodology correct • Re-platform not deviating from purpose of model replacing • Try to avoid learning R ‘on the fly’ and understand good coding • Analytics academy • Coding best practice guide • Lots of style guides

  29. Guides and Tips to Re-platforming • Design • How best to approach the analysis • Will it align with the specification • Testing & QA • Use of R Markdown for QA outputs • Unit testing • Clarity • Is everything clearly explained or a black box • Does it align with documentation • Case studies - BBC’s R cookbook

  30. Key Messages

  31. LUNCH

  32. QA of data science projects Results from the survey

  33. Background • QA Working Group (QAWG) is responsible for the AQUA book and reports to DDANs / Analysis Function Board • Perceived lack of strategic oversight of data science QA • No “big picture” view of how we should run quality assurance • Concern that AQUA does not address data science sufficiently Online survey to take the temperature of the community

  34. Basics of the survey • Publicised on govdatascience Slack site • Circulated to all Heads of Profession in GSG and GORS to cascade to team leaders • We did a bit of chasing with partial responders • Ran from March – June 2019 39 responses from data science teams across government Some departments did not reply (DCMS, MHCLG, DfT, MoD) Findings are not statistically representative – but still useful

  35. Main takeaways • A self-selecting sample of small size – so not ideal. BUT: • QA practice varies widely across teams • Lack of standards and a need for more support was cited as an issue by many respondents • RAP was deployed by some (9 / 39) • It all looks pretty siloed • AQUA is a high level framework – scope to augment with more detailed implementation guidance for data science QA

  36. “What we do now echoes in eternity” What do we do now?

  37. Part 1: Where next for the network? • As a network, what do we need to put in place to make RAP happen as quickly and efficiently as possible? • As a RAP champion, what do you need from the network to make RAP happen in your department? • For each thing, rank it: • RED (this doesn’t exist) • AMBER (this exists, but we need to do something about it) • GREEN (this is working, we have what we need here) And if it’s RED or AMBER, give it a priority to fix (HIGH or LOW)

  38. Part 2: Where next for the network? • For the stuff we’ve come up with: • If it’s RED or AMBER and a LOW priority, we stick it on the “nice to have” list • If it’s HIGH priority we need to fix it – so what how do we, as a network, make it happen? Finally – we prioritise the priorities.

  39. Part 3: Where next for the network? • How should we manage ourselves to deliver what we need? Do we need a formal work programme? If not, how do we manage and review delivery and evaluate success? How do we coordinate activity? Do we need an oversight committee? Task and finish groups?

  40. Next meeting • When should we have the next meetup? • How regular should these be?

More Related