Chapter 8: Quality Management

1. Chapter 8: Quality Management MBAD 7090 IS Security, Audit, and Control (Dr. Zhao)

2. Objectives • Understand quality management • COBIT PO11 • Various standards • System development phases • System development approaches • Audit involvement IS Security, Audit, and Control (Dr. Zhao)

3. Overview • Quality Management (QM) is a process that impacts the effectiveness, efficiency, integrity, and availability of information systems. • A video: QM in 3 minutes IS Security, Audit, and Control (Dr. Zhao)

4. Four Major QM Components • A quality culture/philosophy • Upstream management leadership • Quality assurance • Project management planning and organization • Quality control • Project execution management and administration • Final review and acceptance • Downstream successful project completion IS Security, Audit, and Control (Dr. Zhao)

5. COBIT PO11: Manage Quality • Control objectives • General quality plan • Quality assurance approach • Quality assurance planning • Quality assurance review of adherence to IT standards and procedures • System development life cycle (SDLC) methodology • SDLC for major changes to existing technology • Updating of SDLC • Coordination and communication • Acquisition and maintenance framework for the technology infrastructure • Third-party implementer relationship • Program documentation standards • Program testing standards • System testing standards • Parallel/pilot testing • System testing documentation • Quality assurance evaluation of adherence to development standards • Quality assurance review of the achievement of IT objectives • Quality metrics • Reports to quality assurance reviews IS Security, Audit, and Control (Dr. Zhao)

6. System Development Standards • Capability Maturity Model (CMM) for Software • Developed by Software Engineering Institute (SEI), which was established by US Department of Defense in 1984. • ISO 9000 Quality Management and Quality Assurance Standards • International Organization for Standardization • Six Sigma • Originally developed by Motorola IS Security, Audit, and Control (Dr. Zhao)

7. CMM Maturity Levels IS Security, Audit, and Control (Dr. Zhao)

8. CMM Adoption IS Security, Audit, and Control (Dr. Zhao)

9. CMM and Quality IS Security, Audit, and Control (Dr. Zhao)

10. CMM and Quality (continued) BenefitsLevel 12Level 23Level 34 Reduce Defects 12% 40% 85% Reduce Time 10% 38% 63% Reduce Cost 8% 35% 75% Schedule Variances 145% 24% 25% IS Security, Audit, and Control (Dr. Zhao)

11. ISO 9000 Standards • Set of documents dealing with quality systems that can be used for external quality assurance purposes • ISO 9000:2000, Quality management systems – Fundamentals and vocabulary • ISO 9001:2000 Quality management systems – Requirements • ISO 9004:2000 Quality management systems - Guidelines for performance improvements. • A video • Critics: • The amount of money, time and paperwork required for registration • Prone to failure when a company is interested in certification before quality IS Security, Audit, and Control (Dr. Zhao)

12. ISO 15504 Software Process Improvement & Capability • Goal: assess software processes • A comprehensive reference framework • Designed as a baseline to perform capability determination in an organization • Used when planning, managing, monitoring, controlling, and improving the acquisition, supply, development, operation, evolution and support of software IS Security, Audit, and Control (Dr. Zhao)

13. System Development Life Cycle (SDLC) • Key phases: • Planning: why and how do we build it • Analysis - what we need • Design - how will we build it • Construction - build it • Testing - verify that it works • Implementation - send it out • Maintenance - fix and improve it IS Security, Audit, and Control (Dr. Zhao)

14. Planning Phase • Needs analysis • Current system review • Conceptual design • High level illustration of how the new system will operate • Equipment requirements • Cost/benefit analysis • Project team formation • Project plan IS Security, Audit, and Control (Dr. Zhao)

15. Audit Involvement in Planning Process • Develop an understanding of the proposed system, make sure time is built into the schedule to adequately define controls, and verify that all the right people are involved IS Security, Audit, and Control (Dr. Zhao)

16. Analysis Phase • Users and systems analysts define the system requirements in terms that can be measured • Functionality of the existing system is matched with the new functionality and the requirements are defined and validated with the user • Used as a basis for design phase IS Security, Audit, and Control (Dr. Zhao)

17. Design Phase • Systems analyst defines all system interfaces, reporting, and screen layouts, and specific program logic • Time controls are defined for input points and processing • Screen layouts, controls and reports are reviewed and approved by the user • Used in the construction phase IS Security, Audit, and Control (Dr. Zhao)

18. Construction Phase • Programmer completes the program construction and validates the construction through individual unit testing • The program is tested for both syntax and logic flow • Ensures error routines work IS Security, Audit, and Control (Dr. Zhao)

19. Testing Phase • The system is tested to verify that it works as intended and meets design specifications • Requires an overall testing strategy • Defining the individual test events, roles and responsibilities, test environment, problem reporting and tracking, and test deliverables IS Security, Audit, and Control (Dr. Zhao)

20. Test Event Objectives Unit Testing Unit testing verifies that "standalone" programs match specifications. Test cases should exercise every line of code. Integration Testing Integration testing verifies that all software and hardware components work together. Data is passed from one program to the next. All programs and subroutines should be tested during this phase. Test cases should cover all components (e.g., hardware and software). Functional Testing Functional testing verifies that the application meets user requirements. Test cases should cover screens, navigation, function keys, on-line help, processing, and output (reports, files and screens). Technical Testing Technical testing verifies that the application works in the production environment. Test cases should include error processing and recovery, performance, storage requirements, hardware compatibility, and security (e.g., screens, data, and programs). Acceptance Testing Acceptance testing verifies that acceptance criteria defined during the project definition stage are tested. Test cases should include system usability, management reports, performance measurements, documentation and procedures, training (e.g., users, help desk, production support, operations), and system readiness (Operations/Systems sign-off). Test Events IS Security, Audit, and Control (Dr. Zhao)

21. Implementation Phase • Includes: • Strategy - covers who, what, when, where, and how of the implementation process • Conversion - defines the procedures for correcting and converting data to the new application • Documentation - includes both user and system support procedures • Training - includes end-users, computer operators and maintenance programmers • Support - includes help desk and problem reporting IS Security, Audit, and Control (Dr. Zhao)

22. Training • Users forced to learn on their own take as much as six times longer to become productive • An effective training program reduces support costs by 3 to 6 times because users make fewer mistakes and have fewer questions IS Security, Audit, and Control (Dr. Zhao)

23. Approaches to Software Development • Traditional Information Systems Development (Waterfall) • Purchasing and modifying a packaged system • Prototyping and Rapid Application Development • End-user Development IS Security, Audit, and Control (Dr. Zhao)

24. Waterfall Development • Pros: structure and organization • Cons: lengthy and costly IS Security, Audit, and Control (Dr. Zhao)

25. Prototyping • Pros: quick and intensive user involvement • Cons: • Inadequate analysis and design • May implement the prototype as the final product IS Security, Audit, and Control (Dr. Zhao)

26. End User Development (EUD) • Created, operated, and maintained outside of the IS organization • Limited or no formal procedures • Inadequate controls may exist • Maintenance is solely the responsibility of end user IS Security, Audit, and Control (Dr. Zhao)

27. Case: Western Power quality tool • Please read the handout and answer the following questions: • Please identify and summarize problems in Western Power. • Please discuss possible consequences if Western Power keeps the current practice. • Please provide suggestions and recommendations to help Ms. Smith for her job. IS Security, Audit, and Control (Dr. Zhao)

28. Auditor’s Role in Development Process • One of two roles: • Control consultant – designs application controls into the system • A part of the development team • Independent reviewer – provides recommendations to be acted on, or not, by project management • Not a part of the team IS Security, Audit, and Control (Dr. Zhao)

29. Auditor Key Tasks • Review user requirements • Review manual and application controls • Check all technical specifications for compliance with company standards • Perform design walkthroughs at the end of each development phase • Submit written recommendations for approval after each walkthrough • Ensure implementation of recommendations before beginning the next phase • Review test plans • Present findings to management IS Security, Audit, and Control (Dr. Zhao)

30. Audit Level of Involvement • Determined by: • Completing a risk assessment • Develop an audit plan with specific review points • Communicate scope of involvement and any findings to concerned parties IS Security, Audit, and Control (Dr. Zhao)

31. Risk Assessments • Process risks: • Lack of strategic direction • Lack of development standards • Lack of a formal systems development process • Negative organizational climate • Application Risks: • Application complexity and magnitude • Inexperienced staff • Lack of end-user involvement • Lack of management commitment IS Security, Audit, and Control (Dr. Zhao)

32. Audit Involvement –Post-Implementation • Auditor may survey users to evaluate the effectiveness of the application from a workflow perspective • Review error detection and correction procedures • Perform tests of data to confirm transaction trail • Should not be performed by same audit assigned during the development effort IS Security, Audit, and Control (Dr. Zhao)

33. Change Control • Once in production, changes will be made due to undetected errors or business changes • Should be done in a controlled manner: • Problems are reported, tracked, prioritized, and resolved • Changes are authorized, tested, documented, and communicated IS Security, Audit, and Control (Dr. Zhao)

34. Auditing Quality Assurance • Quality Assurance (QA): planned and systematic production processes that provide confidence in a product's suitability for its intended purpose. • QA activities are planned and documented • Adherence of project activities and products to applicable standards, procedures, and requirements is verified objectively • All impacted groups are aware of and cooperative with QA activities • Noncompliance issues are addressed with senior management IS Security, Audit, and Control (Dr. Zhao)

35. Audit Recommendations • The auditor must always consider the value of the control recommendation versus the cost of implementing the control • Should be specific, identifying the problem and not the symptom • If not implemented, more time and money will be spent in the long run IS Security, Audit, and Control (Dr. Zhao)

36. Audit Report • Interim reports at the completion of major phases: IS Security, Audit, and Control (Dr. Zhao)