ADRI A model for critical analysis

1. ADRIA model for critical analysis A training resource to assist with CDU’s preparation for audit Prepared by A/Prof Martin Carroll PVC Learning, Teaching & Community Engagement September 2010

2. Overarching Principle CDU will be proactive and transparent in its approach to audit. This is an opportunity to support our ongoing quality improvement efforts.

3. What is a Quality Audit? A public and systematic determination of whether: • CDU’s goals and objectives are based on appropriate regulations, standards (note Appendix E) and benchmarks; • CDU’s planned arrangements are suitable to achieve those goals (i.e. the overall approach); • CDU’s actual practice conforms to the planned arrangements (i.e. the deployment); • The arrangements achieve the desired results; • CDU is learning from a self-evaluation of its approach, deployment and results, and can demonstrate improvements.

4. What is a Quality Audit? A public and systematic determination of whether: • CDU’s goals and objectives are based on appropriate regulations, standards (note Appendix E) and benchmarks; • CDU’s planned arrangements are suitable to achieve those goals (i.e. the overall approach); • CDU’s actual practice conforms to the planned arrangements (i.e. the deployment); • The arrangements achieve the desired results; • CDU is learning from a self-evaluation of its approach, deployment and results, and can demonstrate improvements.

5. Reviewing Graduate Destinations ADRI Cycle of Quality Assurance Quality Audit ‘Double Loop’ Process IMPROVEMENT APPROACH RESULTS DEPLOYMENT Internal Review of the System (results in a Portfolio section with Strengths & OFI) External Review of the System (results in Audit Report section with Coms, Affs, Recs) • Evidence examples: • Internal program review reports • Input obtained from industry & professional bodies • Resulting action plans • Reports on progress against actionplans • Evidence examples: • Uni graduate destination targets • Industry needs analysis • Careers Service Plans • Curriculum demonstrating links to industry needs Topic: Graduate Destinations • Evidence examples: • Graduate destinationstatistics • Evidence of employersatisfaction (e.g. surveyresults) • Evidence examples: • Interviews with academic staff,careers service staff, recent graduatesand employers ADRI is a model of quality assurance used by agencies in many countries around the world (e.g. AUQA) It can be applied to any topic – in this example, the analysis of a university’s graduate destinations

6. Key Point Each aspect of ADRI involves consideration of different types of evidence. Only the totality makes sense when analysing the big picture.

7. IMPROVEMENT APPROACH RESULTS DEPLOYMENT

8. Statements of intent (goals, objective etc.) will be found in multiple sources: applicable Legislation Strategic Plan 2010-2014 and internal plans Bylaws, policies, procedures, guidelines Cycle 1 Audit Report MCEETYA National Protocols An unpacking of the Themes External standards ApproachWhat CDU proposes to achieve I A R D

9. Internal and external statements of intent must align. The processes for setting statements of intent require evidence of appropriate benchmarking and stakeholder involvement (this is AUQA’s proxy for “Fitness of Purpose” auditing). ApproachWhat CDU proposes to achieve I A R D

10. Operational Plans – detailing what should be done by when, by whom, to what standard and with what resources. Manuals – detailing how processes should be implemented. Professional development and training aligned to the University’s operational needs. Alignment of resource allocation to plans. ApproachHow CDU proposes to achieve its purpose I A R D

11. What external requirements apply to CDU? How are they incorporated into the Strategic Plan? Does CDU have set of goals, objectives, strategies and targets that are clearly understood by the governing bodies and staff? Was there appropriate consultation, benchmarking and analysis in developing the Strategic Plan? What standards and external benchmarks have been applied? ApproachBroad Questions I A R D

12. Does the planning process incorporate appropriate risk management? Does everyone know what they are supposed to be doing, how and why? Are goals well supported with plans, manuals and training? Are there clear means for monitoring progress against the goals? ApproachBroad Questions (cont.) I A R D

13. An assessment of Approach, on its own, does not tell the whole story – only what is intended. Need to look at a wide range of sources to get the full Approach. Alignment of various goals and strategies ought to be by design, not accident. Equally, alignment of plans with resource allocation models ought to be by design. ApproachCautions I A R D

14. IMPROVEMENT APPROACH RESULTS DEPLOYMENT

15. Also known as ‘implementation’ or ‘process’. Looks at how CDU is implementing its approach. In other words, do the plans and bylaws happen in reality? This is best tested through interviews; tap into people’s ‘lived experiences’. Also includes consideration of input factors such as the quality of resources. DeploymentDeployment Dimensions I A R D

16. What do people do? How do they know if they are doing a good job? Do all staff have the necessary authority and resources to deliver what is expected of them? Do they have the necessary skills and knowledge? Is the organisational structure a help or a hindrance to deployment? DeploymentBroad Questions I A R D

17. Are there appropriate indicators for monitoring the effectiveness and efficiency of processes? How are these reported and used? Are there appropriate means for intervening if necessary? How well do they work? Where the approach is deliberately not being followed, why not? How are changes to the planned processes managed? Are people allowed to contribute ideas? DeploymentBroad Questions I A R D

18. It is insufficient to only focus on deployment. It must relate to an approach and lead to results. It is essential to ‘triangulate’ anecdotal evidence about deployment (e.g. from interviews) with other sources of information, until the issue is ‘saturated’. DeploymentCautions I A R D

19. TRIANGULATE! Corroborate information by obtaining original evidence from more than one type of source Information Source A Topic Information Source B Information Source C SATURATE! Keep collecting evidence until nothing newof significance emerges

20. IMPROVEMENT APPROACH RESULTS DEPLOYMENT

21. Quality cannot be assured by only focusing on the goals, plans, inputs and processes. There must be an emphasis on what is actually achieved – the results! Every intent must have at least one reported result. Every result should link back to a statement of intent. ResultsResults Dimensions I A R D

22. It is essential that a causal relationship can be shown between the approach, the deployment and the eventual result. Otherwise the result may be just chance. If you know that A+B+C = 19 you still do not know what B is (i.e. whether each step in the process is adding value or not). ResultsResults Dimensions 7 + (-6) + 18 = 19 C A B I A R D 7 + (-6) + 18 = 19

23. Results will be both quantitative and qualitative. In order to be meaningfully interpreted, results ought to be expressed as trends over 3-5 years, with targets and benchmarks. Results for many goals will be aggregated from the results of its component objectives. Examples of good practices in reporting results follow... ResultsResults Dimensions I A R D

24. PERFORMANCE MEASURES With result , trend , targets , benchmark averages & best practices Actual Result s Targeted Results Peer Average Upper and Lower Quartiles (Ensurethat the measurement scale is fully understood!) Performance Measure 2006 2007 2008 2009 2010 2011 Annual Result for Goal #1

25. PERFORMANCE MEASURESOne qualitative + quantitative approach Low awareness of issue. Ad hoc plans. Random training. Commitment attained. Planning framework. Training available. Systematic approach. Full set of plans. Training linked to plans. Leading-edge vision. Plans aligned & integrated. Training comprehensive. A Ad hoc practices. Not linked to plans. Not monitored. Numerous good practices. Practice aligns with plans. Processes are analysed. Good practice systemic. Plans inform practice. Monitoring in place. Processes benchmarked. Practice informs plans. Good practices promoted. D Results not linked to plans. Results not measured. Where measured, variable. Most plans have results. Most results reported. Most targets achieved. All plans have results. All results reported. Targets achieved. Stretch targets established. Results are analysed. Targets exceeded. R Staff input limited. Review processes limited. Improvements are random. Staff input allowed. Review framework. OFI and GP identified. Staff input encouraged. Reviews systematic. Benchmarking undertaken. Staff directly empowered. Self-reviews effective. Benchmark host. I Annual Result for Goal #1

26. For each statement of intent, what are the results? Does CDU know exactly how and why those results were achieved? Have these results been appropriately contextualised (e.g. using targets, trends and benchmarks)? What meaning/interpretation does CDU derive from the results? ResultsBroad Questions I A R D

27. It is insufficient to only consider results. Results only make sense in the context of the approach and deployment. The manner in which a result is presented can influence how it is interpreted, so adopt an attitude of ‘healthy skepticism’. That is what the Auditors are trained to do. ResultsCautions I A R D

28. IMPROVEMENT APPROACH RESULTS DEPLOYMENT

29. This dimension looks at what CDU knows about itself in order to get better and better. Intentions should be regularly recalibrated. Processes should get more efficient and more effective over time. Results should indicate increasing success. This requires a comprehensive system of review – not just consideration of results. ImprovementImprovement Dimensions I A R D

30. What data about CDU performance is routinely collected and reported? How is the validity of the data ensured? What happens to the data? How is the Strategic Plan (and other plans) reviewed and revised? What review processes are in place for CDU’s major activities? How does CDU know that the review processes are effective? ImprovementBroad Questions I A R D

31. Is the process of self review, learning and improvement endemic throughout the organisation? Are all staff empowered and encouraged to contribute to ongoing improvement efforts? What has changed/improved as a result of the review processes? What does CDU intend to change/ improve as a result of the review processes? ImprovementBroad Questions (cont.) I A R D

32. If not driven by sound values, this aspect of ADRI can be very threatening and disempowering for staff (who may feel that their jobs are in jeopardy) and students (who may feel that their grades are in jeopardy). Findings from surveys and reviews are often not used effectively. ‘Wet Paint’ syndrome. This is fine – if it leads to improvements. ImprovementCautions I A R D

33. Key Point It is ok to use the audit to catalyse improvements. In fact, it is desirable as evidence of quality improvement in action. But new improvements must be reported as such, and set in an ongoing context.

34. ImprovementCautions I A R D

35. IMPROVEMENT APPROACH RESULTS DEPLOYMENT

36. ADRI Tips This is the overarching method of self review used for AUQA audits – by auditees and by auditors. For any given topic, at any point in time, CDU will be strong in some aspects of ADRI and less strong in others. This is normal. A quality audit looks for evidence of the quality improvement cycle in action. ADRI can be applied at macro and micro levels, but for audit purposes it is best applied at the subsection (specific topic) level.

37. Analysis vs. Problem Solving ADRI is a method for analysing a total QA system. It can be used internally or externally. It can identify strengths as well as opportunities for improvement (OFI). ADRI is not a planning framework, although it provides useful information for planning. ADRI does not identify the best solution to an OFI. To do so would require problem-solving methods (such as benchmarking). Therefore, AUQA recommendations arising from ADRI ought to focus on what needs to be improved, not how it needs to be improved.

38. IMPROVEMENT APPROACH RESULTS DEPLOYMENT Reviewing & improving Approach, Deployment & Results Mission. Planning, designing processes and training staff Analysing measures of outcomes against goals Implementing plans and monitoring processes

39. For further information:www.cdu.edu.au/audit2011Martin Carroll, PVC Learning, Teaching & Community EngagementTel: 8946 6333 Fax: 8946 6199Email:martin.carroll@cdu.edu.auSinead Vincent, Project Support OfficerTel: 8946 7206 Fax: 8946 6928Email: audit2010@cdu.edu.au