Agile Data Standards Development

1. David Carver www.starstandard.org Agile Data Standards Development A 20 minute introduction to agile development with Data Standards

2. 20 Minute Overview • Data Standard Challenges • Traditional Approach • Required Agile Practices • STAR Results

3. Challenges • Changing Requirements • Requirements change • Monthly • Weekly • Daily • Users of the Data Specifications need the standards sooner • Data Specifications typically released every 2 years or longer • Users requirements have changed

4. Challenges • Manual Processes • Schema are created manually one time before release. • Manual Verification of user requirements • Manual Code Merges. No Source Code Control besides Zipping the repository. • Documentation • Maintained outside the schema • Not kept in sync with the schema • Available in limited formats or proprietary formats.

5. Challenges • One-off Implementations • Due to the lack of timely updates to meet end user requirements, users implement one-off implementations. • One off implementations add more work not less when dealing with multiple trading partners. Each partner may have their own requirements, and their own extensions to the schema. • Extensions to schemas should be avoided, and requirements brought back to the organization. • This requires a rapid response to meet the end users need by the standard organization. Release cycles must be reduced to meet the users needs sooner, rather than later.

6. 20 Minute Overview • Data Standard Challenges • Traditional Approach • Required Agile Practices • STAR Results

7. Traditional Approach • Waterfall Project Management Requirements Analysis Development Test

8. Traditional Approach Requirements and Analysis Development T E S T P U B L I S H • Large upfront requirements and analysis • Little time left for Development • Testing usually an after thought or rushed • Publishing saved until the very end, or just before release.

9. 20 Minute Overview • Data Standard Challenges • Traditional Approach • Required Agile Practices • STAR Results

10. Required Agile Practices • Unit Testing • Test Driven Development • XML Schema: Elements, ComplexTypes, SimpleTypes, Code Lists, OAGIS NDR, ATG 2 NDR. • NDR – ATG 2 Version 2 NDR is used as the base. OAGI 9.0 NDR enhances and extends UNCEFACT rules. • A combination of frameworks such as the NIST NDR Validation framework, Schematron, and JUnit XML Unit testing frameworks should be leveraged • If available leverage existing test beds and suites through out the development process and build process. • Run these as often as possible in an automated fashion.

11. Required Agile Practices • Continuous Integration • Source Code Control • All Development checked in daily • XSDs, Ant builds, Sample XML, Unit Tests • Unit tests and other validations should be run every time source is rebuilt. • All tests and source compilation is automated, no manual intervention. • Guarantees the same steps are followed every time.

12. Required Agile Practices Cont. • Iterative Development • Iterations 2 Weeks in Length. • Requirements, Unit Tests, Schema Development • Publication ready build at end of iteration. • Automated Builds • Automate everything! • Builds, Guidelines, Testing. • Tools need to work how you want them too, not enforce a methodology on you. • Involve the Stakeholders • Those requesting the requirements need to be involved throughout the process • Open and Honest communication is a must.

13. 20 Minute Overview • Data Standard Challenges • Traditional Approach • Required Agile Practices • STAR Results

14. Unit Testing Stats STAR 5 Unit Tests • Fields • 1702 • Components • 881 • Nouns • 296 • BODs • 504 STAR 4 Unit Tests • Fields • 1281 • Components • 360 • Nouns • 252 • BODs • 222 Estimated Total Tests: 5,498 Estimated Runtime: 20 minutes Tests are run every Milestone, and Integration

15. STAR's Results • Publish time in hours.

16. STAR's Results • Building the Repository and Guidelines in hours.

17. STAR Results • Manual Process replaced with Automated Processes • Errors caught within hours instead of weeks or months. • Could publish repository nightly. • Only published 4 times previously • Elimination of Knowledge Silos • More accurate schemas meeting members needs sooner.

18. STAR Lessons Learned • Unit Test..Unit Test...Unit Test • Catches Errors Early • Provides a Regression Test Suite • Run as often as possible • Catches changes that break compatibility • Automate Everything Possible • Have tools work for you, not enforce a methodology on you. • Conduct Nightly, Milestone, and Release Builds • Unit Test..Unit Test...Unit Test • USER ACCEPTANCE TESTS MUST COME FROM THE USERS.

19. www.starstandard.org David Carver - dcarver@starstandard.org Michelle Vidanes - mvidanes@starstandard.org