1 / 51

IS2010 Model Curriculum Core Assessment Exam

IS2010 Model Curriculum Core Assessment Exam . Supporting Institutional and Professional Development Lynn McKell – Brigham Young U Kewal Dhariwal – Excelsior College & ICCP Tom Hilton – U Wisconsin (Eau Clare) Bruce White – Quinnipiac U. Agenda. Introduction/Background – McKell (moderator)

berget
Télécharger la présentation

IS2010 Model Curriculum Core Assessment Exam

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. IS2010 Model CurriculumCore Assessment Exam Supporting Institutional and Professional Development Lynn McKell – Brigham Young U Kewal Dhariwal – Excelsior College & ICCP Tom Hilton – U Wisconsin (Eau Clare) Bruce White – Quinnipiac U

  2. Agenda • Introduction/Background – McKell (moderator) • ICCP Certification Council – Dhariwal • IS2010 Model Curriculum Assessment – Hilton • ISA Examination • Institutional Assessment/Improvement – White • Individual Certification – McKell • ISA Exam Procedures – Dhariwal • Questions

  3. Background • ICCP - Org. 1971 by AITP (DPMA), ACM, etc. • Professional certifications governed by industry associations • ISECON 2001 panel • Possible role of ICCP exams in curriculum assessment • Conclusion: A new assessment instrument was needed • IS2002 Model Curriculum – Exam Development • 2002 Workshop : ICCP and U of South Alabama • Representatives from 18 Institutions • NCC 2003 Calibration • Uses: • Institutional Assessment/Improvement (Accreditation) • Individual Assessment (Certification: ISA) • IS2010 Model Curriculum • Exam Update in process

  4. ICCP Certification Council(*K) • Members: Academic & Industry Professionals • Exam Item creation/validation • Benchmarking and Vetting • Item Statistics and Tracking • Item updating and retiring • Future: ABET, SIGITE

  5. ICCP Certification CouncilMembers • John Whitehouse Jr., PhD, CCP • Director of Certification (Cleveland, OH) • Rick Seaman, VP Financial systems (Virginia) • Terry Linkletter, Professor (Seattle) • David Horvath, Lecturer, Author (Philadelphia) • Lynn McKell, PhD, Professor (Provo) • Pat Cupoli, DAMA International, (Philadelphia) • Jim Woolen, CIO, PhD, Professor (Michigan) • Bob Grenier, PhD, Professor (Illowa, IW) • Tom Hilton, PhD, Professor (Eau Claire, WI) • Bruce White, PhD, Professor (Hamden, CT) • Kewal Dhariwal, PhD, Professor (Albany, NY) • Ken Bainey, PhD, CIO, Author (Edmonton, AB) • Many (>40) others who have contributed to the examinations

  6. Item Creation and validation • Exam Writer’s guide • Complexity of items • Blooms taxonomy of learning objectives • Depth • Subject outline classification • Items reviewed by the Certification Council before being included into the ICCP test bank • Items’ updating and retirement practice • ICCP test banks are conservatively valued between $7-$9million • 29 examinations available across a broad spectrum of information and communications technologies (ICT)

  7. Benchmarking and Vetting • Alpha Exam development • ACM, AIS , IEEE and AITP – Model Curriculum Standards • Subject outline developed (Test Management Council) • Individual Subject Matter Experts develop test items (SMEs) • Group of reviewers modify, wordsmith and classify items to a subject outline, set initial complexity level. • Beta exam is offered to select colleges with students • Beta results analyzed, Items assessed and updated if needed, scores for beta exam takers adjusted if needed • Reporting system finalized • Examination released to the market

  8. ICCP Test Maintenance System

  9. Statistics on exam performance • EXAM – Information Systems • Version=2009304, • Number of exam takers=1026 • Age=0.64 • Pass rates • Practitioner (>50%) = 83.3%, • Mastery(>70% ) = 15.1% • Range of Scores • Low 30, • High 82 • Mean=59 • Standard Deviation=9.900

  10. Item level statistics • Difficulty of items • Hardest 12% get correct • Easiest 99% get correct • Point BiSeRial PBSR Correlation of item with Mastery level exam passers • Lowest -0.05, low discriminant item • Highest 0.48), high discriminant item • Split Half Rho=.77 • KR20=.79 • Skew=-.53 • Kurtosis=-4.20

  11. Comparing different Versions of the exam • V=2009304, • N=1026 • Age=0.64 • Pass(P,M)=(83.3%,15.1%) • Range=(30,82) • Mean=59 • SD=9.900 • Dif=(0.12,0.99) • PBSR=(-0.05,0.48), • Split Half Rho=.77 • KR20=.79 • Skew=-.53 • Kurtosis=-4.20 • V=2007118, • N=1498 • Age=3.15 • Pass(P,M)=(81.7%,20.0%) • Range=(2,85) • Mean=60 • SD=11.405 • Dif=(0.16,0.99) • PBSR=(-0.05,0.48), • Split Half Rho=.80 • KR20=.85 • Skew=-.49 • Kurtosis=-4.07

  12. Item Level Statistics • Item Number 8325 • Creation date: February 2009 • Author: Certification Council • Number of exam takers: 1623 • Percentages choosing{A,B,C,D,Skip}: {.15 .06 .20 .59 .00}, correct - D • PBSR correlation with mastery level exam passers {ABCD}: (-.13 -.25 -.14 .33)

  13. Item 8325 Performance

  14. Exam Presenter

  15. Marking the Exam – Score

  16. Student - Performance Profile

  17. IS2010 Model Curriculum Assessment: ISA Structure(*T) • Model Curriculum Structure • “The structure has become more flexible and adaptable to local conditions while keeping a national/international alignment” – Hilton, 2010 • Model Curriculum Learning Objectives • ISA Examination Content • ISA Examination Structure and Administration

  18. IS 2002 Review IS Minor IS 2002.P0 Personal Productivity w/ IS Technology I S 2002.1 Fundamentals of Information Systems http://www.acm.org/education/is2002.pdf IS 2002.2 E- Business Strat, Architecture, & Design IS 2002.3 Information Systems Theory & Practice IS 2002.4 IT Hardware & System Software IS 2002.5 Programming, Data, File & Object Structures IS 2002.7 Analysis & Logical Design IS 2002.6 Networks & Telecomm IS 2002.8 Physical Design & Implementation with DBMS IS 2002.9 Physical Design & Implementation in Emerging Envs IS 2002.10 Project Mgt & Practice

  19. IS 2010 Structure I S 2010.C1 Foundations of Information Systems http://www.acm.org/education/curricula/IS%202010%20ACM%20final.pdf IS Minor IS 2010.C2 Data & Information Management IS 2010.C3 Enterprise Architecture IS 2010.C4 IS Project Management IS 2010.C5 IT Infrastructure IS 2010.C6 Systems Analysis & Design IS 2010.C7 IS Strategy, Management & Acquisition IS 2010.E1 Application Development IS 2010.E2 Business Process Management IS 2010.E3 Collaborative Computing IS 2010.E4 Data Mining / Bus Intelligence IS 2010.E5 Enterprise Systems IS 2010.E6 Human-Computer Interaction IS 2010.E7 Information Search & Retrieval IS 2010.E8 IT Audit & Controls IS 2010.E9 IT Security & Risk Management IS 2010.E10 Knowledge Management IS 2010.E11 Social Informatics

  20. IS 2010 Learning Objectives Reaching Beyond the Business School Information Systems“Technology-EmpoweredBusiness Development” • Business • Business

  21. IS 2010 Learning Objectives Reaching the Engineering/Health / other domains Information Systems“Technology-EmpoweredEngineeringDevelopment”

  22. IS 2010 Learning ObjectivesKnowledge and Skills of IS Graduates

  23. IS 2010 Learning ObjectivesInformation Systems Specific Knowledge & Skills • Identify and design opportunities for IT-enabled organizational improvement • Analyze trade-offs in solution & sourcing alternatives • Design and implement information systems solutions • Manage ongoing information technology operations

  24. IS 2010 Learning ObjectivesFoundational Knowledge and Skills • Lead and collaborate • Communicate • Negotiate • Think analytically, critically, creatively, ethically • Apply statistics and probability

  25. IS 2010 Learning ObjectivesKnowledge & Skills per Domain Fundamentals • General models of the domain • Business models • Business process design and management • Organizational theory • Business strategy • Key specializations within the domain • Finance • Accounting • Marketing • Operations management; service science and management • Organizational behavior • Business law • Evaluation of performance within the domain • Analysis of organizational performance • Analysis of individual and team performance • Business analytics • Business intelligence

  26. ISA Examination StructureCore Examination • Foundations of Information Systems • Data and Information Management • Enterprise Architecture • IT Infrastructure • IS Project Management • Systems Analysis & Design • IS Strategy, Management & Acquisition • 258 Randomly Selected Items • In Two 105-Minute Sessions with a 10-Minute Break • At Program-Offering Institution

  27. ISA Examination StructurePossible Elective Examinations • Application Development Available • Business Process Management Available • Enterprise Systems Available • Human-Computer Interaction Future • IT Audit and Controls Integrated • IS Innovation and New Technologies Future • IT Security and Risk Management Available • Knowledge Management Future • Social Informatics Future • 110 Randomly Selected Items per Examination • In One 90 minute Session per Examination • At Program-Offering Institution

  28. ISA Examination Structure Subject Outline • 1.0 Foundations of Information Systems • 1.1 Information Systems in Organizations • 1.2 Valuing Information Systems • 1.3 Information Systems Components and Infrastructure • 1.4 Systems Development and Acquisition • 1.5 The Internet and World Wide Web • 1.6 Security of Information Systems • 1.7 Business Intelligence • 1.8 Globalization • 2.0 Data & Information Management • 2.1 Database Approach • 2.2 Data Modeling • 2.3 Database Languages • 2.4 Data and Database Administration • 2.5 Transaction Processing • 2.6 Data / Information Architecture • 2.7 Data Security Management • 2.8 Data Quality Management • 3.0 Enterprise Architecture • 3.1 Enterprise Architecture Frameworks • 3.2 Monitoring and Metrics for Infrastructure and Business Processes • 3.3 Green Computing • 3.4 Business Continuity

  29. IS2010 CORE cont’d • 4.0 IT Infrastructure • 4.1 Systems Concepts • 4.2 Operating Systems • 4.3 Networking • 4.4 IT Control and Service Management Frameworks (COBIT, ITIL, etc.) • 5.0 Project Management • 5.1 Introduction to Project Management • 5.2 The Project Management Lifecycle • 5.3 Managing Project Teams • 5.4 Project Initiation and Planning • 5.5 Managing Project Scheduling • 5.6 Managing Project Resources • 5.7 Managing Project Quality and Risk • 5.8 Systems Procurement • 5.9 Project Execution, Control and Closure • 6.0 Systems Analysis & Design • 6.1 Business Process Management • 6.2 Structuring of IT-based Opportunities into Projects • 6.3 Analysis and Specification of System Requirements • 6.4 Implementation Strategies • 6.5 Design and Analysis • 7.0 IS Strategy, Management & Acquisition • 7.1 The Role of IS in the Organization • 7.2 Structuring the IS Organization • 7.3 Acquiring Information Technology Resources and Capabilities

  30. ISA Exam Structure 3-level outline (small sub-set) • 1.0  Foundations of Information Systems(20%) • Information systems are an integral part of all business activities and careers. This section covers the key components of information systems - people, software, hardware, data, and communication technologies, and how these components can be integrated and managed to create competitive advantage. • 1.1  Information Systems in Organizations 1.1.1. Characteristics of IS professionals 1.1.2. IS Career Paths 1.1.3. Cost, Value and Quality of Information 1.1.4. IS and Organizational Strategy 1.1.5 Value Chains and Networks • 1.2  Valuing Information Systems 1.2.1. Making a Business Case for Information Systems 1.2.2. Multi-Criteria Analysis · 1.2.3. Total Cost of Ownership and Return on Investment 1.2.4. Purchasing Systems and IT Infrastructure 1.2.5. Identifying and Implementing Innovations

  31. Institutional AssessmentAnd Improvement(*B) • Analytical Reports • Closing the Loop • Useful in Accreditation

  32. Institutional Assessment • Some Background: • The Information Systems Management (ISM) program at Quinnipiac University is accredited by ABET and is an AACSB accredited School of Business • We do have an assessment program incorporating several data items • For our most important direct assessment, we have used the Information Systems Assessment test since 2004

  33. Direct Assessment Test • IS Assessment Test • Taken as part of ISM 440 Project Management, our undergraduate capstone class. • While the IS Assessment test has three main sections, we concentrate on sections 2 and 3: • 2.0 Organizational and Professional Skills (which maps to our Role of IS in Organizations goal) • - 2.1 Individual and Team Interpersonal Skill • - 2.2 Business Fundamentals • 3.0 Strategic Organizational Systems Development with IS • - 3.1 Organizational Systems Development • - 3.2 Project Management

  34. Additional Information • ABET expects ‘direct’ measurement of learning. This test gives us the best analysis • It also helps us identify areas of weaknesses and places to improve. (see next pages) • We encourage students to take the test seriously by making it part of their final grade

  35. And … down to specific learning units

  36. Analytical reports

  37. ABET Goals ABET Program Educational Outcomes • Objectives and Assessments • INTENT • That the program has documented, measurable objectives, including expected outcomes for graduates. The program regularly assesses its progress against its objectives and uses the results of the assessments to identify program improvements and to modify the program’s objectives. • STANDARDS • I-1. The program must have documented, measurable objectives. • I-2. The program’s objectives must include expected outcomes for graduating students. • I-3. Data relative to the objectives must be routinely collected and documented, and used in program assessments. • I-4. The extent to which each program objective is being met must be periodically assessed. • I-5. The results of the program’s periodic assessments must be used to help identify opportunities for program improvement. • I-6. The results of the program’s assessments and the actions taken based on the results must be documented.

  38. ABET Program Outcomes 2010-2011 • An ability to apply knowledge of computing and mathematics appropriate to the discipline • An ability to analyze a problem, and identify and define the computing requirements appropriate to its solution • An ability to design, implement, and evaluate a computer-based system, process, component, or program to meet desired needs • An ability to function effectively on teams to accomplish a common goal • An understanding of professional, ethical, legal, security and social issues and responsibilities • An ability to communicate effectively with a range of audiences • An ability to analyze the local and global impact of computing on individuals, organizations, and society • Recognition of the need for and an ability to engage in continuing professional development

  39. ABET Program Outcomes cont’d • An ability to use current techniques, skills, and tools necessary for computing practice. • Coverage of the fundamentals of • i.1 a modern programming language • i.2 data management • i.3 networking • i.4 data communications • i.5 systems analysis and design • i.6 the role of Information Systems in organizations • i.7 project management • An understanding of processes that support the delivery and management of information systems within business application environments

  40. ABET Program Outcomes Report

  41. Individual Certification: ISA • IS Model Curriculum Exam • Managed/Maintained by the ICCP • Individual Performance Measure • Basis for Information Systems Analyst (ISA) certification • Several other certification requirements • Proficiency Levels • Sample of other ICCP Certifications: • Associate Computing Professional (ACP) • Certified Computing Professional (CCP) • Certified Data Management Professional (CDMP)

  42. ISA Exam Procedures(*K) • Examination Administration Procedure • Application form • Proctor agreements • USB or networked USB • Cost to college: $50/student to a max of $2,000 per college • Cost to student: $250 to receive their ISA credential upon graduation from their degree and must join recertification program • Results Reporting • Unofficial student performance profile • Office student performance profile • Institutional reports • National and Local scores by subject outline • National and Local scores by IS2010 Courses and learning units • Students ranked by order of performance best to worst, by college • ABET Program Outcomes report on request • Points of Contact • ICCP office : 800.843.8227; email: office@iccp.org website: www.iccp.org

  43. INVITATION: • Adopt the ISA exam at your institution • Use it in curriculum development • Use it in the accreditation process • Help create a database to support development of national/international norms. • Encourage certification

More Related