A Validation Methodology for Graphics Processors

1. A Validation Methodology for Graphics Processors Thesis Work Presentation 10.10.2006 Author: Valtteri Rantala Supervisor: Prof. Raimo Kantola Instructor: M. Sc. Harri Syrjä

2. Content • Background • Objectives & methodology • Graphics processors & design process • Validation in general • Validation methodology • Validation of the Vector Graphics unit • Conclusions • Future study

3. Background • ATI technologies Finland (formerly Bitboys oy) designs IP cores for mobile devices • Quality is extremely important in processor design • Errors are extremely costly to repair after the design is implemented in silicon • Customer dissatisfaction • Company saw it necessary to reinforce the quality process • Validation was a part of the quality assurance that needed improvement

4. Objectives & methodology • Objectives • To develop the validation process for graphic processors • To specify documentation for the validation process • To find suitable testing techniques to be used in the validation process • Scope • Only validation aspect of quality assurance is inspected • Only validation of one unit is inspected in the thesis • Individual test cases are not introduced • Methodology • Literature survey • Discussion with experts

5. Graphics processors & design process • Graphics processors • Used to accelerate the calculation tasks in different stages of the 3D pipeline • Specialised in calculating computer graphics algorithms • Contains a set of arithmetic units that are design for specific tasks • The structure of the graphics processors is based on the use of units Direct-X 3D-pipeline Illustrative structure of a graphics processor

6. Graphics processors & design process • Design process for graphics processors • Graphics processor roadmap definition • Defines what kind of products are to be developed • Core specification • Feature set, architecture, size, performance, quality requirements • Unit specifications • Usage and operational details • Validation • TLM – Transaction Level Model • Verification • The models are verified against each other and in some cases against the specifications • RTL – Register Transfer Layer Design flow of a graphics processor

7. Validation in general • Validation is a part of the quality assurance • The goal of the validation is to discover defects as early as possible • Validation is defined as “The process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements”(IEEE1012) • Consists of analysis, evaluation, review, inspection and testing of products • Different validation approaches • Walk-through • Formal methods • Test development • Prototyping

8. Validation methodology • Validation process was proposed • The validation process is divided into five different phases • Product Validation Plan • Unit Validation Plan • Test Case Design and Procedures • Validation Report • Validation Feedback • Validation sub process is executed for each unit • Each phase has input and outputs documents • Each phase has specific tasks

9. Validation methodology • Product Validation Plan • To employ the validation resources efficiently, to monitor, to control, and to allow the identification of each participant’s role and responsibility • The purpose of this document is to describe the validation program for the product on a high level • Validation Program Management • Describes the program schedule, milestones and goals for each milestone, defect and reporting management, and staffing and risk management • Concept Level Validation • Describe on a system level what the system must be able to do • Performance, power consumption, acceptance use cases and APIs • Hardware Level Validation • Describes feature level validation • Hardware features are introduced on a high-level and the overall validation methodology for each feature group is described

10. Validation methodology • Unit Validation Plan • For every unit • Describes how the specific unit will be validated • Contains to tasks: feature analysis and validation goal analysis • Feature analysis • All the features that are to be implemented are analysed • All features must be: clear, unambiguous, internally consistent and testable • Priority class is given for each feature • Validation goal analysis • features are analysed and the validation goals for each feature is presented • Feature spreadsheet is made • Target feature, validation items, methodology description, number of test cases

11. Validation methodology • Test Case Design and Procedures • All testing work is done in this phase • Test case execution • Result analysis • Defect reporting • Test Case Design Principles • Test cases should uses the same coding standards as product developers • Only to test a feature • Size is 128x128 pixels or smaller, RTL is 100 to 1000 times slower to run than TLM designs. • Test Execution Framework • Test environment • Validation environment

12. Validation methodology • Test environment • All test cases are executed in test environment • Contains three implementations: Reference, TLM, RTL • Each implementation generates a bitmap image • Third party software implementation can be used as a reference • Compiles test cases and executes them in hardware models • Command stream input is used to hold register level data, which is used as an input to hardware models

13. Validation methodology • Validation environment • The fully automated validation environment • Execute tests • Compare images from different sources • Produce a report from executed test cases Validation environment Report structure

14. Validation methodology • Defect Reporting • Defect reporting system was used (JIRA) • Centralized location, defect count, reporting, issue handling • The following information should be available for each defect • Defect Id, date of detection, product/project id, defect criticality • Location of the defect (file name, code line if possible) • The version of the file/design/release • Description of the defect • Name of the test case showing the defect • Proper reporting = faster fixing times Defect handling diagram

15. Validation methodology • Validation Report • The purpose of this phase is to give a good understanding of the current validation status • Readership is all stakeholders • Generated weekly basis -> produces history data • Validation report contains the following information: • Status tracking of various variables • Test case implementation progress • Test cases passing rates for different design models • Defect tracking results • Test case coverage • The goal is to show that all the features are tested • Code coverage analysis • Statement coverage is used • Coverage for each file

16. Validation methodology • Validation Feedback • The purpose is to get information about how well the process has worked • The Feedback is analysed and recommendations for improving the validation process are done based on the feedback • Collecting Feedback • From the process results • Low failure rate/high pass rate, defect density, inconsistent results • To make questionnaires and interview different stakeholders • Change Management • To have a planned approach to change implementation • Major/minor changes

17. Validation of the Vector Graphics unit • Validation process was used to validate the Vector Graphics unit • All validation process phases and tasks were executed • The feature analysis revealed a few errors in specification documentation • Test case design based on the unit specification • The test case principles guide the designing of the test cases • Code coverage analysis revealed unspecified features • Quality issues: what is accurate enough?

18. Conclusions • Importance of the validation was realised • Written process for the validation of the graphics processor with documentation templates • Predictability of the validation process was improved due the validation reporting and proper planning • No major rework was made in the project • Defects found in TLM after validation dropped from 30% -> 5% • There is now real history data to be used in the future projects • The documented progress increased confidence towards the product

19. Future study • The work for improving the validation process will continue after this thesis • The areas for future work are the areas that received most negative feedback • The use of validation metrics and the use of history data must be stressed in future projects • The quality of test vectors will be stressed in future projects

20. Any questions?

21. Thank you!