Where’s the Data? Conducting a First-Year Data Audit

Where’s the Data?Conducting a First-Year Data Audit Karen Paulson Asheville, North Carolina – July 2002

The Data Audit Toolkit • Developed Through a Partnership • The Policy Center on the First Year of College • John Gardner, Betsy Barefoot, Randy Swing, Mike Siegel, and Marc Cutright • NCHEMS • Karen Paulson and Peter Ewell • Generously Supported by: • The Pew Charitable Trusts • The Atlantic Philanthropies

Pilot Institutions • Augustana College (IL) • Blue Ridge Community College (VA) • Lynchburg College (VA) • Northeast State Technical Community College (TN) • Ohio University • Santa Fe Community College (FL) • The University of Texas at El Paso • University of Cincinnati • University of Minnesota Duluth • Washington State University

Data Audit “The process of identifying data resources and uses wherever they may be within an institution and gathering them into a useable information system.” NCHEMS Administrative Rationale manual, page 15

Basic Premise I • Most of the Data Needed for Meaningful Analysis Already Reside in Institutional Data Systems • Admissions • Registration/Transcripts • Student Surveys • Assessment Databases • Individual “Service Office” Records

Basic Premise II • The Major Task Is: • To Organize These Data in Useful Ways • To Make Access and Analysis as Flexible and Straightforward as Possible

The Data Audit Process • Is NOT Itself an End • Is a Means for Understanding and Improving the Use of Data and Information on Campus  Key to Improving the First Year of College

“…we tend to assume that all first-year programs were implemented as planned and that the experiences of all students were uniform.”

In Reality There Are Three First-Year Experiences* • The “Official” One (Designed and Published Plans) • The “Delivered” One (Actual Institutional Actions) • The “Experienced” One (Reality Including Student Choice) * Adapted from Joan Stark

The Main Question…. • How can your institution analytically disentangle the many elements of the first college year and provide evidence about the effectiveness, or lack thereof, of related first-year programs, policies, and procedures?

Some Preliminary Questions • What Is Unique About the First Year of College at Your Institution? • What Is it Like to Walk in a Student’s Shoes at Your Institution? • How Many Surveys, Tests, or Assessments Does a Student Really Experience During the Year? • Does Any Information from These Data Collections Get Communicated Back to the Students? How? • How Are These First-Year Data Used by Faculty and Administrators?

A Quick Self-Assessment • What Are the Primary Sources of Data About Students, Curricula, and Programs—Specifically Related to the First Year of College—Currently in Place at Your Institution? • Are Any of These Underutilized? Why? • What’s Missing About Which You Would Like Information?

Why a Data Audit? • Identify and Inventory Data Sources • Identify and Inventory Data Needs • Support Assessment of... • What Happened • What Mattered • Foster a Culture of Data Use

Primary Data Audit Activities The Supply Side Conducting a Campus-Wide Examination of Existing Data Sources The Demand Side Determining Which Data Are Most Needed for Evaluation, Assessment, and Decisionmaking

Key Components of a Data Audit • Identifying Sources of Data • Inventorying These Data Sources • Compiling Information About Data Sources • Identifying Gaps in Data Sources • Assessing What Data Users Need and Want to Know • Determining Which Analyses (Existing or Suggested) Have Utility for Stakeholders of Your Institution or Unit

Organizing a Data Audit The Data Audit Leadership Team • Institution-Wide Committee • IR, Student Affairs, Academic Affairs, Advisors, Orientation Staff, Faculty, Registrar Staff, Students, etc. • Team Members • People Directly Involved with the Institution • People Directly Involved with Data • First-Year Professionals not Normally “Data Connected” • Leader(s) with Broad Campus Support and Respect

The Right Attitude • Fresh Perspective • Open Attitude • Collaborative Approach Not a “Gotcha” Mentality

Visiting Potential Data Sites • Reasons for Physical Visits • Honors Unit Personnel on Their Turf • Builds Relationships • Allows You to Read Reactions • Allow You to Do Immediate Follow-Up and Collect Artifacts • Discovers Hidden Databases

Types of Data • Official Data • Registrar Systems • Admissions • Institutional Research Offices • Assessment Offices • “Unofficial” (Guerrilla) Databases • Unit Records Kept in Local Computers • Viewed as “Single” Purpose/Disposable Records

Supply Side Questions • Types of Data? • Who Collects Data and Why? • How Complete Are Data? • Where Do Data Go? • “Walking the Process”

Supply Side Questions to Ask • What Kinds of Records or Data Do You Keep on First Year Students? • How Complete Are Data Collected? • How Are Data Entered? • What Schedules Govern Your Data Collection? • How Are Data Updated?

Finding Existing Data • Follow Student “Footprints” • Time-Based Investigation • Who Collects Data? • Structures-Based Investigation • Why Are Data Collected? • Functions-Based Investigation

Supply Side: Items to Collect • Actual Forms and Questionnaires • Data Element Dictionaries • Data Element Definitions • Database Structures and File Formats

Demand Side Questions • Who Are the Key Constituencies? • What Are the Existing Reports and Requirements? • To Whom Are Data and Information Reported? • What, and When, Are the Decision Cycles? • How Current and Accurate Does the Information Need to Be? • What Are the Gaps in Existing Data?

Demand Side Questions to Ask • To Whom Do You Report Data and Information? • What Information Do You Need or Wish You Had? • What Are the Gaps in Data? • How Are Data Used By Others?

Demand Side: Items to Collect • Representative Reports to Constituents • Copies of External Data Reporting Requirements

Looking for... • Completeness of Data Gathered • Availability of Data • Integrity of Data • Consistency of Data Definitions • Who Coordinates Data Collection? • Who Coordinates Use of Data? • Who Controls Data and Data Processes?

Data Audit Output • Synthesized and Coherent Picture of Existing Data • Data Element Lists • Data Definitions • Data Locations and Locus of Responsibility • Data Collection Timetables • Plan for Common Data Collection/Sharing • Identification of Unmet Data Needs

Design of Recommended Data Structure • Define Core Data Elements • Responsibilities and Data Flows • Core Indicators and Calculational Routines

Data Leverage Points • Problem Identification • Persistence Rate • Violations of Prerequisite Sequencing • Student/Advisor Ratios • Context Setting • Achievement by Gender or Race • Patterns of Student Performance • Entering Student Characteristics • To Inform Discussion • Overall Patterns Rather than Anecdotal or Single • Selling Decisions • Creating Buy-In for Action

Using Evidence to Stimulate and Manage Change • Start with Obvious Discrepancies • Between: • Perceptions and Reality • Designs and Delivery • Expectations and Results • Among Different Constituencies and Stakeholders • Avoid “Perfect Data Fallacy” • Recognize There Will Always Be “Errors” • Use Triangulation and Multiple Indicators • Know How Good Is Good Enough • Use Information in “Layers” • Avoid Excessive Complexity • Package Data Around Problems • Disaggregate as Needed in Response to Questions and Opportunities • Use Information to Start Discussions, not “Give Answers” • Develop Multiple Interpretations • Establish “Data Dialogue”

The Data Audit Toolkit(coming fall 2002) • Administrative Rationale • A non-technical overview and rationale for a data audit. • Technical Manual • A detailed guide, listing data elements and data, as well as step-by-step plans for a data audit.

Thank You! • Karen Paulson, 303.497.0354, Karen@nchems.org • Mike Siegel, 828.877.6009, siegelmj@brevard.edu If you have questions or want to discuss the data audit and analysis project further, please contact:

Where’s the Data? Conducting a First-Year Data Audit