“Grown-ups like numbers…” The Role of Quality Assurance in Higher Education

1. “Grown-ups like numbers…”The Role of Quality Assurance in Higher Education Stephan Neetens Bologna Process Expert - Flemish Community of Belgium SSU Conference, Rogaška Slatina, Slovenia “Quality Higher Education: Here, There, Anywhere?” 30/11/07-01/12/07

2. 1. Overview (1) 1. Overview 2. Introduction 3. History of quality assurance 4. QA concepts from quality management science 5. Societal context of quality assurance in higher education 6. Goals of quality assurance in higher education 7. Definition of quality 8. Structural approaches to quality assurance 9. Assessment levels 10. Assessment subjects

3. 1. Overview (2) 11. Assessment organisation 12. Challenges 13. Conclusion

4. 2. Introduction (1) • J. Huizinga, How Does History Determine the Present? A Discourse that Was Never Held, Leiden, 1946: “And now I haven’t even touched upon the most difficult word in our sentence: the concept how? How, that means: in which manner. How poses the insoluble question about the quality of things. The quality now, even of the smallest thing, escapes the means of our logical thinking. Applied to our theme how immediately inverts the question: how does history determine presupposes the the more profound question: how is history itself determined, and this means: by what is history determined, and the by what immediately changes into a by whom and only this question finds an answer, but outside the borders of the logical mind: by God, and this answer reveals immediately that the how is incomprehensible, because it is God’s business. With all our philosophy and science we once more end up at the modest warning Dante paraphrased Aristotle with: State contenti umana gente al quia. Be satisfied, human race, with mere existence.”

5. 2. Introduction (2) • If questions about quality are really insoluble and really defy logical thinking, than the emphasis on quality and quality assurance in HE policy poses a very big problem. • If questions about quality unavoidably lead us to God, than the policy of quality and quality assurance are a specialty of the philosophy of God or even worse of dogmatic theology. • Although questions about absolute quality are Byzantine, I would like to treat them as a provocative challenge and say to the HE community: “Non state mai contenti, gente dell’accademia, al quia.”

6. 3. History of quality assurance (1) • QA originated in world of manufacturing and industrial production • before Industrial Revolution (<1780): quality = straightforward problem -> production centralised in 1 person’s hand who performs and controls -> standards for products with external inspection system (guilds, city governments, customs officials,…) e.g.Book of the Eparch, Constantinople, 912 -> QA in the process sense poses no explicit problem

7. 3. History of quality assurance (2) • Industrial Revolution (> 1780): -> collectivisation of production process leads to mass production whereby no longer 1 individual performs and controls the production -> structural introduction of foreman, but mainly for organisatory and command purposes (military-hierarchical organisation of labour process) • throughout 19th C.: mass production leads to low standard products, widely diverging output, a lot of defective products -> quality becomes an explicit problem linked to the organisation of the production process

8. 3. History of quality assurance (3) • end 19th C: in US 1st forms of explicit QA introduced as part of larger industrial management reforms: -> F.W. Taylor (1856-1915): Taylorism: scientific management combined with quality departments to oversee quality of production and rectify errors -> H. Ford (1863-1947): Fordism: standardisation of design and component standards ensure production of standard products, combined with quality departments and inspection of the output • 1930s: 1st application of statistical controll as a QA method

9. 3. History of quality assurance (4) • WO II = very important impetus for QA in US and UK: need to raise both quantity and quality of production for war effort • after WO II: quality and QA becomes both profession, management process and scientific discipline • clear evolution in QA: -> simple quality control: 40s-60s -> quality engineering: 70s -> quality systems engineering: 90s • no longer limited to manufacturing, but also widely applied in practically all sectors of service provision (banking, health care,…) e.g. today 31% of companies with ISO 9001 certificate are service providers

10. 4. QA concepts from quality management science (1) • in quality management science throughout the last 50 years all kinds of concepts, methods, tools and strategies conceived e.g. SPC, Zero Defects, Six Sigma, quality circles, Total Quality Management (TQM), Theory of Constraints (TOC), Quality Management Systems (QMS), continual improvement (Kaizen),… • mostly originated in manufacturing but also widely applied in service provision -> also in education • 3 concepts regularly encountered in QA world in HE: PDCA-cycle, ISO 9001 standards, TQM

11. 4. QA concepts from quality management science (2) 1) PDCA-cycle: • Plan-Do-Act-Check-cycle or Shawhart-cycle or Deming-cycle • developed by W.A. Shawhart in 1930s in Bell Telephone Laboratories • made famous by W.E. Deming, father of modern quality control • 4 elements: • PLAN: establish the objectives and processes necessary to deliver results in concordance with specifications • DO: implement the processes • CHECK: monitor and evaluate the processes and results against objectives and specifications and report the outcome • ACT: apply actions to the outcome for necessary improvement, which means reviewing all steps (P-D-C-A) and modifying the process to improve its next implementation

12. 4. QA concepts from quality management science (3)

13. 4. QA Concepts from quality management science (4) 2) ISO 9001 standards and certificate: • example of Quality Management System (QMS) • QMS: • set of policies, processes and procedures required for planning and execution (production/development/services) in the core business area of an organisation • integrates the various internal processes within an organisation and intends to provide a process approach for project execution • enables an organisation to identify, measure, control and improve the various core business processes that will ultimately lead to improved business performance • Quality Management System Standards of the International Organisation of Standardisation (ISO) • first issued in 1987, as the translation on international level of the British industrial management standard BS 5750 developed during WO II

14. 4. QA concepts from quality management science (4) • aims at certifying the processes and the system of an organisation and not the product or service itself • originated in manufacturing, but now employed across a wide range of other types of organisations • updated every odd year: last version = 2000, next version = 2008 • many HEI’s (especially in the non-university sector) have received a certificate of compliance with the ISO 9001 standards

15. 4. QA concepts from quality management science (5) • version 2000 meant a radical change by going away from product inspection and preventive paperwork to the concept of process management -> examples of standards: • set of procedures that cover all key processes in the business • monitoring of processes to ensure that they are effective • adequate record keeping • checking of output for defects, with appropriate corrective action where necessary • regular review of individual processes and the system itself for effectiveness • facilitation of continual improvement

16. 4. QA concepts from quality management science (6) 3) Total Quality Management (TQM): • originates in the work of A. Feigenbaum, Total Quality Control (1951): • Total Quality Control defined as: “An effective system for co-ordinating the quality maintenance and quality improvement efforts of the various groups in an organisation so as to enable production at the most economical levels which allow for full customer satisfaction.” • quality control goes from a technical management process to a business method

17. 4. QA concepts from quality management science (7) • TQM = management strategy aimed at embedding awareness of quality in all organisational processes • combination of 3 elements: • total: involvement of the entire organisation, supply chain and/or product life cylce • quality: fitness for purpose • management: system of managing with steps like plan, organise, control, lead, staff, provision,… • clear parallel with the quality culture discourse in HE (cf. EUA-project)

18. 5. Societal context of quality assurance in higher education (1) • massification of HE since the beginning of 70s in Western-Europe and beginning of 90s in Eastern Europe: • e.g. Flanders: 1958 -> 5000; 2008 -> 170000 • educational concepts did not adjust (e.g. Humboldian university ideal) • funding did not raise at same rhythm • quality of education went down • QA is instrument to raise quality based on reasoning that funds must not only be spent but spent efficiently -> within current budget efficiency gains can be made • change in relationship between government and HEI’s: • growing legal and practical autonomy for HEI’s: freedom to organise programs and determine content, to appoint staff, to spend budget, to determine profile,… • government demands post factum control via QA system

19. 5. Societal context of quality assurance in higher education (2) • bigger role for private provision of HE: • private profit and non-profit HE providers(especially in Eastern Europe) • transnational education (especially in Eastern Europe) • QA as an instrument for market entry regulation and consumer protection • influence of corporate life and economic science: • development of quality control as an applied economical science • wide application of quality management strategies in profit and non-profit service sector • new vision on relation between government and the institutions/agencies it subsidises • from inspection to assessment/audit • from only controll to also consultancy • from organised distrust to guaranteed trust

20. 5. Societal context of quality assurance in higher education (3) • European Union policy context: • also in HE is Europe becoming an ever more closer union: art. 149-150 EC, EC mobility programmes, Bologna Process, Copenhagen Process,… • integration of and exchange between nationale HE education systems require mobility of students and staff and recognition of degrees and study periods • QA is an instrument to support mobility and create the institutional trust necessary for smoother and quicker recognition • growing international competition between HE systems and HEI’s -> QA is an instrument to make national HE systems and its degrees more readable and understandable, and more trustworthy abroad (quality labels)

21. 5. Societal context of quality assurance in higher education (4) • the fall of the Berlin Wall and the crumbling of the communist bloque: • disavowment of (neo)marxist ideology • Undermining of belief in state planning and collective action on state leve lto tackle collective action problems • embracement of a retreating and at arms length government concept • reinforcement of the faith in self-regulation

22. 6. Goals of quality assurance in higher education (1) • QA can have one or more of the following goals: • quality improvement • quality control/consumer protection • public accountability • information provision • international comparison/benchmarking • ranking • funding • recognition

23. 6. Goals of quality assurance in higher education (2) • the last 2 goals can be seen as indirect goals: the first 6 is what a QA system is explicitly geared towards, the last is what the government or a government agency uses the outcome of QA for • the goal or goals of QA determine the set-up and functioning of the QA system, e.g.: • the role of a self-evaluation report in QA for improvement >< QA for control • the use of reference points in QA for improvement >< the use of standards in QA for control • the publicity status of evaluation reports in QA for improvement >< QA for information • QA for ranking requires comparable criteria and standards and provisions to facilitate transversal analysis

24. 7. Definition of quality (1) • traditional description: something is of good quality when it does what it is meant to do -> poses the same problems as Aristotle’s well-known definitions of justice: • to give each person what is due to him • to treat like cases alike and unlike cases unalike BUT what is due to a person and when are cases alike and when unalike, and above all who determines this? -> what is something meant to do and who determines this? • 2 major definitions of quality in HE possible: • fitness for purpose • fitness of purpose

25. 7. Definition of quality (2) • fitness for purpose: • relates to traditional description • relates to world of economics and quality management science • something is effective and efficient: it does what it is meant to do in the most profitable way possible • = procedural or relative description

26. 7. Definition of quality (3) • fitness of purpose: • beyond traditional description • beyond world of economics and quality management science • asks the questions what is good education, what is good programme • raises pedagogical discussions: e.g. what is best: problem-based learning? collaborative learning? ex cathedra learning? • raises content discussions: e.g. what does a good legal education entail? : • next to technical legal courses also a broad introduction into other fields of social sciences? • next to traditional legal methodology also an active introduction into the methodology of other fields of social sciences, so as to create an interdisciplinary approach to law (e.g. Law & economics)? • clinical legal education and/or moot court exercises? • = absolute or material description

27. 7. Definition of quality (4) • possible to formulate general competences across fields of study for HE graduates (cf. TUNING project) • in some fields it is easier to formulate domain-specific competences than in others (e.g. chemistry >< history) • in the end the absolute description is also relative: • peers sit together and determine what fitness of purpose is for a specific programme at a specific point in time • years later the fitness of purpose can and maybe must have changed, taken into consideration scientific evolutions in the field of study and evolutions in society • M. Weber: “Ideas and concepts change when old men die.”

28. 7. Description of quality (5) • definition of quality used in a QA system is important for the set-up and methods of the system: • fitness for purpose: focus on management evaluation methods and techniques and the instruments and statistics needed for these • fitness of purpose: trustworthy development of a list of standards or reference points necessary • good and balanced QA in HE pays attention to both fitness for and of purpose

29. 8. Structural approaches to quality assurance (1) • 4 ways of organising QA: • evaluation • accreditation (certification) • benchmarking • audit

30. 8. Structural approaches to quality assurance (2) • evaluation: • traditional approach to QA in HE • a group of peers with an open ended agenda comes in, looks at aims, processes and output, judges and formulates recommendations • linked to quality improvement function • accreditation: • an independent instance determines if a programme or institution fulfils a list of standards that is fixed on beforehand and the same for all programmes or institutions taken into consideration • linked to quality control and accountability functions

31. 8. Structural approaches to quality assurance (3) • benchmarking: • programmes or institutions are compared to each other on strong and weak points • this can be done on the basis of reference points fixed on beforehand • if the reference points are conceived of as standards this can lead to ranking of programmes or institutions • linked to (international) comparison (and ranking) function(s) • audit: • integrated approach to quality and quality management • focuses very strongly on internal processes and procedures for policy steering, problem detection and problem solving • usually combined with a focus on institution or faculty, not programme • linked to quality improvement function

32. 8. Structural approaches to quality assurance (4) • every approach has its own specific consequences for set-up and methodology • this analysis makes clear that the traditional dichotomy and antinomy between QA and accreditation fake is: • QA and accreditation are not opposites, accreditation is one of the forms of QA differing on the basis of aim and set-up • when people oppose QA and accreditation, what they really mean is the difference between evaluation and accreditation • it has been observed during the past 10 years that evaluation and accreditation are growing towards each other:

33. 8. Structural approaches to quality assurance (5) • accreditation is paying more and more attention to quality improvement and strategies to induce it e.g. Hungarian accreditation system, big US debate on the future of accreditation, the conception of the Dutch-Flemish accreditation-system • evaluation is making more and more use of reference points to make judgements more objective, consistent and comparable e.g. subject benchmarks in the UK QA system, indicator lists in the Flemish QA system

34. 9. Assessment levels (1) • 4 possible levels for external QA: • programme • logical cluster of programmes/faculty • institution • HE system

35. 9. Assessment levels (2) • programme: • advantage: direct view and influence on quality of education provision on the basic level • disadvantage: for many quality issues a programme depends on wider policy it cannot influence let alone control (e.g. funding, staff, counseling, facilities, housing,…) • example par excellence: the so-called Dutch model (1985) • institution: • advantage: you have a systems approach, you can focus on how an institution as a whole deals with quality issues, which processes and procedures are in place, if there is a quality culture, and you can assess the wider policy that influences the quality of individual programmes • disadvantage: no direct check of the real life quality of individual programmes • example par excellence: the French system (1984)

36. 9. Assessment levels (3) • logical cluster of programmes/faculty: • usually not considered to be a full-fledged option, but a compromise alternative to overcome the disadvantages of QA on programme and on institution level • quite rare in practice • HE system: • not really an alternative • if done, on a meta-level: e.g. how do institutions tackle QA? • mainly used as policy input for parliament and government

37. 9. Assessment levels (4) • end 1990s fierce debate between proponents of programme QA and proponents of institution QA: • proponents of programme QA mainly students, governments, parliaments -> quality improvement and accountability • proponents of institution QA mainly institutional leaders and their umbrella organisations (e.g. EUA) -> stress on institutional autonomy, trust and responsability

38. 9. Assessment levels (5) • in practice the border lines are once more much more blurry: • systems based on institutional QA are looking for ways to include the programme level (e.g. France, UK post 2001); systems based on programme QA are looking for ways to include the institutional dimension (e.g. Sweden) -> there seems to be growing a consensus that a good QA system combines programme and institution evaluation • evolutionary theory: in a first stage programme QA is indispensable, after eliminating the rotten apples and through this processing stimulating the HEI’s to build strong internal QA systems external QA can shift focus from programme to institution (BUT probably a rationalising apology for British policy failure) • combination strategy: in the US system all HEI’s have to be institutionnaly accredited, but next to the you have a lot of programme accreditation in those fields of study where society has an interest to guarantee the quality of individual programmes and graduates (e.g. doctors, lawyers, nurses, teachers, engineers,…)

39. 10. Assessment subjects (1) • QA in HE can assess the following subjects: • education • research • facilities (libraries, laboratories, counseling services,…) • Management • QA of education can assess the following subjects: • input • process • output

40. 10. Assessment subjects (2) • input: • aims and goals of the programme • content of the programme • number of staff • Staff qualifications • number of students • staff/student ratio • facilities (library, laboratories, ICT-infrastructure,…) • …

41. 10. Assessment subjects (3) • process: • educational concept • teaching method • work forms • forms of evaluation • studeability of the programme • complaints procedure • internal QA system • …

42. 10. Assessment subjects (4) • output: • retention/success rate • level of master thesises • employment rate • time lapse before 1st employment after graduation • wage level of graduates at 1st employment and after 5 years of experience • satisfaction level of alumni • …

43. 10. Assessment subjects (5) • traditionally the focus in QA is on input, but this gives a distorted image: process and output have to taken into consideration, especially output, because this indicates the level of attainment of a programme • focus on output in QA links up with the concept of learning outcomes in curriculum development and evaluation -> QA procedures should try to assess in an objective and systematic manner if programme graduates attain the learning outcomes their programmes put forward for them

44. 11. Assessment organisation • in the Berlin Communiqué (2003) the European ministers responsible for HE agreed on a brief codification of best practice for QA organisation: an national QA system must be based on the following principles: • main responsibility lies with HEI’s: they develop an internal QA policy and system • clear definition of responsabilities of all involved institutions and organisations • evaluation of programmes and/or institutions containing a.o. internal self-assessment, external evaluation, student participation and publication of results • system of accreditation, certification or comparable procedures • international participation, cooperation and networking • further developed in the European Standards and Guidelines for QA in the Bergen Communiqué (2005)

45. 12. Challenges • how to guarantee that QA also improves quality? • how to avoid a compliance culture? • how to avoid the stifling of educational experiment and innovation? • how to diminish the administrative burden? • how to keep costs down? • how to organise international cooperation and participation? • how to build a European roof on national systems so that they can communicate and exchange?

46. 13. Conclusion (1) • QA is a necessary good because it guarantees that university professors deal with education in a manner as professional as they deal with research • QA is a necessary good because it gives students the lever to become actively involved in the quality management of their programmes while at the same time respecting the academic freedom and content expertise of their teachers • the major challenge for the future is to gear QA systems more towards quality improvement because only then will they be worth their money and burden -> a.o. this means widening the focus of QA: not only numerical checking but also content and policy debate and recommendations

47. 13. Conclusion (2) A. de Saint-Exupéry, The Little Prince: “Grown ups like numbers. When you have a new friend they never ask you: ‘What is the colour of his eyes? What sounds does he like? What games does he like to play?’ No, instead they ask you: ‘How old is he? How big is he? What grades does he get in school? How much does his father earn?’ By knowing numbers they think they know the person. Ah, but you shouldn’t hold it against them. You must be very patient with grown-ups.”