Data Collection: Overview and Design Matrix

Data Collection: Overview and Design Matrix Research Methods for Public Administrators Dr. Gail Johnson Dr. G. Johnson, www.researchdemystified.org

Steps in the Research Process Planning 1. Determining Your Questions 2. Identifying Your Measures and Measurement Strategy 3. Selecting a Research Design 4. Developing Your Data Collection Strategy 5. Identifying Your Analysis Strategy 6. Reviewing and Testing Your Plan Dr. G. Johnson, www.researchdemystified.org

Planning the Data Collection Strategy If you don’t know where you are going, you can wind up anywhere Dr. G. Johnson, www.researchdemystified.org

Data Collection Strategy • Consists of three inter-related steps • First: what data collection methods make sense given: • What you want to know • Numbers or stories? • Where the data resides • Environment, files, people • Resources available Dr. G. Johnson, www.researchdemystified.org

Data Collection Strategy • Second: how much structure is needed? • Quantitative • Use when you want to do statistical analysis, want to be precise, know exactly what you want to measure and/or want to cover a large group • Qualitative • Use when you want anecdotes or in-depth information, when you are not sure what you want to measure, and/or there is no need to quantify Dr. G. Johnson, www.researchdemystified.org

Data Collection Strategy • Third: how much data to collect? • Identify the population of interest • Decide whether to collect data from the entire population or a sample? • If a sample • How large a sample? • Random or non-random • Sampling will be presented later Dr. G. Johnson, www.researchdemystified.org

Quantitative Approach • Highly structured: check-a-box format • Closed-ended questions • Precise measures every time • Reliable: same exact measures every time • Unforgiving of mistakes • Harder to develop than open-ended questions • Easier to analyze than qualitative data • computer will easily crunch the numbers Dr. G. Johnson, www.researchdemystified.org

Qualitative Approach • Open-ended questions, fill-in-the-blanks • Easier to develop than closed-ended questions • Forgiving of mistakes • Often yields “rich”, i.E. Lots of data Dr. G. Johnson, www.researchdemystified.org

Qualitative Approach • Labor intensive to gather • Harder to analyze: • Time-consuming with many possible meanings • Risk of bias in interpreting • We may unconsciously see things that are not really there but fit our viewpoint Dr. G. Johnson, www.researchdemystified.org 9

In Practice: A Mix Is Often Used • A research project might combine several approaches, including both quantitative and qualitative: • Available data, interviews and a survey • Survey and a focus group • Available data with observations Dr. G. Johnson, www.researchdemystified.org

First: Where Does the Data Reside? • Does the data already exist? • Where are the best sources of data? • Who collected the data? • How reliable is the data? • Can you get access to it? Dr. G. Johnson, www.researchdemystified.org

First: Where Does the Data Reside? • Do you have to collect new data? • Is the data in the environment? • Is it in a filing cabinet or computer database? • Is it in news stories? • Is it in legislative language? • Is it with people? Dr. G. Johnson, www.researchdemystified.org 12

Your Local School: Where Is This The Data? • Number of students • Number of teachers • Educational background of teachers • Number of books and other learning resources • Number of graduates • Knowledge and skills learned by students • Satisfaction of parents Dr. G. Johnson, www.researchdemystified.org

Consider Operational Definitions And Trade-offs • Operational definitions: • Do students who get GEDs count in graduation rates? • Which data might be easier to obtain? • Which data might be harder to obtain? • Trade-offs often have to be made • Data collection limitations must be acknowledged Dr. G. Johnson, www.researchdemystified.org

Data Collection: General Guidelines • Data collection must be consistent • Establish the plan and rules and stick to them • Use available data if already exists. • Faster, cheaper, easier • But find out how they: • Collected the data • Defined the variables • Ensured accuracy of the data Dr. G. Johnson, www.researchdemystified.org

Data Collection: General Guidelines If you must collect original data: • Establish procedures that are consistently followed • Maintain accurate records of definitions, coding • Pre-test, pre-test, pre-test, pre-test • Verify accuracy of coding, data input Dr. G. Johnson, www.researchdemystified.org

Design Matrix: A Planning Tool • A tool that helps focus on details • It is a visual • Focus is on content not writing style • It is a living document • Many changes before it is final • This is generic format • Change it to fit your style Dr. G. Johnson, www.researchdemystified.org

Design Matrix: A Planning Tool • Basically tracks the planning steps • Research question and sub questions • Likely measures • Likely design • Likely sources of information • Likely data collection strategy • Likely sampling choice Dr. G. Johnson, www.researchdemystified.org

Design Matrix Form Change as Needed Dr. G. Johnson, www.researchdemystified.org

Design Matrix • The planning process is iterative. • Researchers go round and round before all the details are nailed down. • Most research projects have several questions. • The plan should be specific for each question. • Work on one question at a time. Dr. G. Johnson, www.researchdemystified.org

Design Matrix • Leave blanks, fill in as more information becomes available. • Use comments to identify areas that require more information. Dr. G. Johnson, www.researchdemystified.org

Completing The Puzzle • All pieces of a research design should fit together • It is a process: early ideas and assumptions change as new information comes to light • Test all data collection instruments and plans before going live • Practice collecting the data in real setting • Expert review • Cold reader review Dr. G. Johnson, www.researchdemystified.org

Design Matrix: Tool For Assessing Research • Every research article can be broken apart into its component pieces in a design matrix • Makes it easier to see to see what they actually did • Be wary of disconnects • Systematically breaking the research into pieces shatters the spell of convincing narrative • Remember: research flaws are almost always in the planning components • Measures, design and data collection Dr. G. Johnson, www.researchdemystified.org

Takeaway Lessons • While data collection is presented here as a single step in a linear process, in practice, it is interwoven with decisions about the measures and sampling • Data collection requires tremendous attention to detail • No statistical wizardry will save mistakes in data collection • It takes more time than anyone expects Dr. G. Johnson, www.researchdemystified.org

Upcoming:Data Collection Toolkits • Available data • Observations • In-person interviews • Focus groups • Surveys: mail, telephone, email, and internet surveys Dr. G. Johnson, www.researchdemystified.org

Creative Commons • This powerpoint is meant to be used and shared with attribution • Please provide feedback • If you make changes, please share freely and send me a copy of changes: • Johnsong62@gmail.com • Visit www.creativecommons.org for more information Dr. G. Johnson, www.researchdemystified.org

Data Collection: Overview and Design Matrix