1 / 20

SA2 - Quality Assurance

SA2 - Quality Assurance. Alberto Aimar (CERN) SA2 Leader 1 st EMI Periodic Review Brussels, 22 June 2011. Outline. Objectives Achievements KPIs Lessons Learned Outlook. SA2 in EMI. NA1, NA2. SA2. Requirements. NA1, JRA1. Software & Services. Defines. Quality Assurance Process

leal
Télécharger la présentation

SA2 - Quality Assurance

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SA2 - Quality Assurance Alberto Aimar (CERN) SA2 Leader 1stEMI Periodic Review Brussels, 22 June 2011

  2. Outline • Objectives • Achievements • KPIs • Lessons Learned • Outlook 1st EMI Periodic Review

  3. SA2 in EMI NA1, NA2 SA2 Requirements NA1, JRA1 Software & Services Defines Quality Assurance Process Policies Metrics Reviews Tools Infrastructure Support Process definition SA1 JRA1 SA2 Certifies Implements Process monitoring JRA1 SA1 Release Candidate 1st EMI Periodic Review

  4. SA2 QA Objectives Define and establish a common software quality assurance process and metrics for all engineering activities. Enable a continuous integration and testing process by selecting and maintaining tools and resources for building and testing software either within the project of in collaboration with external resource providers Allow the EMI middleware to consistently pass the customer acceptance criteria and continually improve the software quality and the process itself by monitoring the metrics value trends, reviewing quality control activities and related tests, providing support and consultancy in QA matters. [DoW] Processes and Metrics • Establish common software quality assurance process and metrics Tools and Testbeds • Provide tools and resources for building and testing • Support a continuous integration and testing infrastructure Monitoring and Review • Monitor metrics trends, reviewing and improving software quality Customers QA Criteria • Allow passing customer acceptance criteria User Support on QA • Support and consultancy within the project

  5. SA2 QA Strategy EMI is merging 4 established Middleware projects, each with its own QA practices • Work with a 3-years vision Y1: explain, define and implement Y2: review and automateY3: consolidate and optimize • Use established QA standardsISO /IEC 9126-1-4 and 25000 • Benefit from existing QA practices Use existing QA tools, resources and expertise 1st EMI Periodic Review

  6. SA2 Management Strategy Strategy applied for all SA2 activities and tasks • Involve PTs, SA1, JRA1 - Merging 4 products, tools, platforms, etc is a challenge,we involved all teams. • Provide early solutions- We wanted to provide early policies, tools, repositories, etc because teams were waiting for them • Explain QA constraints- Make aware all developers and teams that QA will requires changes for converging on common solutions • Be transparent– Work closelywith other activities (SA1, JRA1, EMI EMT Product teams, EGI QA, PO), all SA2 material public • Aim at automated solutions – They provide exponential benefits compared to any specific or manual solutions • Support tools and testbed– Not just policies and rules to follow, but also give useful infrastructure and services to the developers 1st EMI Periodic Review

  7. SA2 Information https://cern.ch/twiki/bin/view/EMI/SA2 (bit.ly/emisa2) 1st EMI Periodic Review

  8. Objective: Common QA Process Define and establish a common software quality assurance process and metrics for all engineering activities [DoW] SO 1.4 Software Quality Assurance Plan defined early • SQA factors, SQA management • Roles and procedures • Standard Practices and conventions Detailed QA Policies • Release and Change Management • Configuration and Integration • Packaging and Distribution • Testing and Certification • Documentation SA2 EMI QA policies are used by all Product Teams to achieve a uniform process and presentation of EMI software, documentation, etc 1st EMI Periodic Review

  9. Some QA Policies 1st EMI Periodic Review

  10. Objective: Common QA Metrics Continually improve the software quality and the process itself by monitoring the metrics value trends [DoW] SO 1.4 Selected Metrics to objectively qualify status of software • Automated numerical metrics • Cover different requirements of EMT, SA1, PT, SA2 • Provide reports, trends and dashboards Generate a wide range of metrics to cover several QA aspects • Metrics on code, process, documentation, support • Internal and external metrics (code and process) • Language dependent (Java, C++, Python), object-oriented Examples of metrics • Reaction time to critical bugs, delays in releases • SLOC, code complexity, bug density, test coverage Automated tools to collect the same QA metrics across all EMI products. QA data generated to be analysed & key metrics selected 1st EMI Periodic Review

  11. Objective: Provide EMI QA Tools Enable a continuous integration and testing process selecting and maintaining tools and resources for building and testing software [DoW] SO 1.5 Initial situation: All different tools, dev environments, trackers, etc Final Goals: Move towardsa single tool set, workflow and data Approach Followed • Define common software engineering workflow - Uniform build, testing, packaging trying not to affect the developers’ efficiency • Provide build and test infrastructure - Provide a comprehensive infrastructure able to support the policies (build, packaging, releasing, etc). This infrastructure is ETICS. • Introduce progressively testing, QAmetrics – provide automated test execution, metrics generation and reporting • Focus on Support - Support integrated in GGUS, 50% of users were new to the tools All EMI software is built, packaged and distributed with the same tools and is all following same policies 1st EMI Periodic Review

  12. Tools - Initial Situation 1st EMI Periodic Review

  13. Tools - Current Status XML Exchange Format 1st EMI Periodic Review

  14. Objective: Common QA Testbed Enable a continuous integration and testing process either within the project or in collaboration with external resource providers [DoW] SO 1.4 Created the EMI Testbedfor integration and testing process • Single Testing Infrastructure within Quality Assurance • Product Team – centric model for their individual certification Support Internal Inter-Component Testing • Focus on interaction among different clients and components provided by other Product Teams and for release candidates verifications Defined Large Scale Testing infrastructure • Scalability and interoperability testingon real environment • Involved external customers, resources and their verifications (EGI QA) Installed all EMI Services both production and release candidates Agreed participation of EGI sites to large scale testing All EMI Software (~90 installations) production and release candidates on EMI testbed (7 sites) for testing & certification 1st EMI Periodic Review

  15. SA2 – Year 1 Requirements NA1, JRA1 SA2 Software & Services Defines Process definition SA1 JRA1 Quality Assurance Process Policies Metrics Reviews Tools Infrastructure Support SA2 Certifies Implements Process monitoring JRA1 SA1 Release Candidate In Y1 SA2 established the foundations of a measurable QA process based on common policies, tools and infrastructure 1st EMI Periodic Review

  16. Outline • Objectives • Achievements • KPIs • Lessons Learned • Outlook 1st EMI Periodic Review

  17. SA2 KPIs 1st EMI Periodic Review

  18. Lessons Learned • Merging QA for four products all with different processes, tools is really complicated, technically and humanly • Strong support of all partners was crucial for adoption • Combine top-down QA constraints & bottom-up needs • Involve teams experts and whoever want to contribute. Constantly explain the goods of shared common solutions • Start with simple metrics, same for all, explain and discuss details. Select metrics suitable for different audiences • Provide early tools and testbedinfrastructure with useful services in order to support the policies and the process • Periodically review and adapt keeping in mind the overall QA goalsof the project and developers needs 1st EMI Periodic Review

  19. SA2 in EMI Year 2 Main QA Focus for Year 2 (in addition to Y1 activities) • Review and adapt QA policies, tools, testbed after Year 1 • Select Key Metricsand work with PTs to improve on them • Distribute standard QA reports automatically generated • Provide SL6 and Debian 6 new platforms on ETICS, Testbed • Improve on Unit Testing PTs should add tests in ETICS • Add Tests on testbed PTs provide automated tests • Progress with Large Scale Testbed with volunteer EGI Sites Management Approach for Year 2 • Follow the QA strategy with same approach used in Y1Worked very well. Y2: review and automate. Involve PTs in QA matters in order to have their support 1st EMI Periodic Review

  20. Thank you EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611 1st EMI Periodic Review

More Related