Analysis Does findings really magically emerge?
Basic Steps in Data Analysis Review proposal and research question Scan (fill gaps if needed) Check data for completeness Jot notes Write summaries Speculate Determine categories/identify themes Notice relationships Theorize and eliminate rival theories (interpretations) Begin, often, with typologies Try to confirm and to disconfirm interpretations Adapted from LeCompte & Preissle (1993), pp. 234-254 Suggestions for Data Analysis Collect and write up data Read through data (at least twice), looking for commonalities and agreements. List themes and variations on themes. Study or introduce variations to sharpen understanding of the phenomenon. Seek other information to help explain the phenomenon (e.g., research literature, novels). Consider how your newfound understanding will improve your practice. Based on Barritt, Beekman, Bleeker, & Mulderij (1985). Researching Educational Practices, ch. 3. University of North Dakota: Center for Teaching and Learning. Vague descriptions
Types of Analysis • Based on your goal (i.e., QL method) • Ethnographic analysis • Focus on culture and social regularities of everyday life • Classification scheme could emerge from the data themselves or be preexisting category schemes • Narrative analysis • The study of experiences through stories • Analyze the text using techniques of a particular discipline
Types of Analysis • Phenomenological Analysis • Get at the essence or basic structure of the phenomenon • A variety of techniques • Content Analysis and Analytic Induction • Content analysis can be quite quantitative in nature -- look at frequency and variation of messages. Simultaneously code raw data and construct categories • Analytic induction – discrepant or negative case analysis. “constant refinement of hypotheses as the researcher finds instances that do not match the original hypotheses. Eventually a hypothesis evolves that explains all known cases of the phenomenon. The object is to achieve a perfect fit between the hypothesis and the data” (Merriam, p. 160, emphasis added).
Types of Analysis • Constant Comparative Method • Is compatible with most inductive focus of all QL research and therefore can be adopted by many who do not seek to build substantive theory
Levels of Analysis • Level 1: Descriptive account. • Just describe what was – a narrative. [see Mrs. Schneider’s classroom – Day 1] • Level 2: Category construction • Construct categories or themes that capture recurring patterns that cut across the data set. • Level 3: Making inferences, developing models, generating theory • More than a category scheme – look at relationships among the categories so to explain some aspect of educational practice and allow researcher to draw inference about future activity.
Analysis as Category Construction “Picture yourself sitting down at the dining room table, ready to being analyzing data from your modest qualitative study. In one pile on your left there are three hundred or so pages of transcripts of interviews [you get the picture] . . . Now what do you do? Where do you start? . . . You begin by reading a transcript, and then another. You realize you should have asked the second participant something that came up in the first interview. You quickly feel overwhelmed; you begin to feel that you are
Don’t let this be you! literally drowning in the data. It is doubtful that you will be able to come up with an findings. You are undermined your entire project by waiting until after all the data are collected before beginning analysis” (p. 161).
Analysis as Category Construction • Day 1 -- first data entry (observation/interview/document) • Review the purpose of your study. • Read, reread the data, making notes in the margin. • “What kind of notes?” you ask. • Anything that strikes you as interesting given the purpose of your study. • Write a memo that captures your “reflections, tentative themes, hunches, ideas, and things to pursue that are derived from your first data set” – things to ask, observe, or look for in your next data collection activity. [see Mrs. Schneider’s classroom – Day 1]
Analysis as Category Construction • Day 1 or Day 2 – next data entry (observation/interview/document) • Do the same thing as with your first day entry • Compare the first with the second. • Make more notations, create more memos, make more plans for what to look for, ask about, etc.
Analysis as Category Construction • The process continues on and on and on! • When you have gathered your data, you will have a list of tentative categories/themes from which to approach your data. • Then what? Read your data – all of it – perhaps a number of times, doing just as you have done.
Analysis as Category Construction • Read through your earlier notes and memos. • Begin to apply the categories to the entire data set (find instances that reflect the category). By this time, some of the elements will already be labeled because of your early analysis, but other things will need to be relabeled given the continuing development of your view of the data.
Documenting Category Development • Create a master list • Document (by date and perhaps by what data has been reviewed) versions of the master list. A master list should include • Name of the category • A definition of what is included in the category, and perhaps also what is not included in this category. • Exemplars of the category. • Versions of the list will be important if you publish an article – seems that reviewers who know about QL research are always curious about how the categories evolved. Keep notes on what happens to your categories – why some drop out, how others are expanded, and how some are combined. • [See constructivism study – category evolution] Next slide
Naming a Category • How do you come up with the name of a category? • From the literature (make sure that what you choose is compatible with the purpose and theoretical framework of the study) • From the researcher (you decide) • From the participants (often they provide very good descriptors that can be used as a category name) Return to previous page
Characteristics of Categories • As a group your categories should • Be exhaustive – all important data should be contained in them. • Be mutually exclusive in that a unit of data only fits into one category (this is Merriam’s view, I happen to disagree). • Be sensitizing – should be as sensitive as possible to what the data contains that you are trying to capture. • Should be conceptually congruent – with the same level of abstraction (again, perhaps I disagree with this one – different categories may serve different purposes in the end).
How Many Categories? • As many as you need! • In other words, it depends on the data. • The number should be manageable – too many is a problem. Too many may lead to a discrete description.
Examples • Knowledge construction studies • Constructivism
Guidelines for Categories that are Comprehensive & Illuminating • If a number of informants mention it, then it may be an important dimension. • Audience (who are you trying to inform about the study) may determine what is important. • Unique categories may stand out and should be retained. • Some categories may reveal areas of inquiry that might not otherwise be recognized. • From Lincoln & Guba • There should be a minimum of unassigned [interesting/intriguing] data.
Check Your Category Coding • Very important for continuity in your coding, particularly if you have coded over a long period of time. • Read through all of the data elements in one category. Do the elements seem to belong together? Do they work with the category definition that you have created? Do you need to update the definition? Do you need to move some data to another category? • [see Native/Non-native Speakers in Face-to-Face/Online learning environments, stop before findings notes]
Check Your Categories • Graphical display • Display your categories in graphic from • Write your purpose statement at the top of the form. • Ask yourself – are the categories answers to your research questions.
Multiple coders (IRR) • Using another coder can help develop trustworthiness in your interpretations • You do not have to calculate IRR – there are potential problems involved in calculating IRR in some QL data analysis. • What should you do?
Holistic Data Analysis: Guiding Questions • What are the important things we’ve learned? • What are we overemphasizing? • What are we neglecting? • What are the things we want to say? • Do we have strong enough data to say this? • What other data do we need? • Have we attempted to disconfirm? • What’s the big picture? • What are the crucial points or aspects? • What are the crucial relationships among points? • What facilitates / enhances / supports? • What obstructs / undermines / complicates?
Level 3: Making Inferences, Developing Models, Theorizing • The category scheme does not tell the whole story! • Look across your data – Do/how do these categories work together? (this is largely an element from grounded theory, but I think it’s useful process in moving beyond descriptive accounts). If you have a coding system that is not mutually exclusive you may actually find some leads by the multiple codes that help you identify these relationships. What does it all mean????? • [see Native/Non-native Speakers Processing Notes] • (note: we are omitting some of Merriam’s further description – because we are not developing theory).
Assertions • The single-sentence distillation of an important research finding. • Is difficult to develop • Generally do at the end. • Features • Rarely longer than a single sentence. • Address important or pervasive findings, not trivial matters • The strongest possible statement about what was learned that can be supported by the data collected and presented. • Include qualifications that limit the scope of interpretation to the data collected and presented.
Assertions • Assertions can address the research question; in most studies, one or more assertions will do so. But assertions can also state incidental findings unanticipated by the research question. • The quality of a research study is not determined by the number of assertions it yields. The quality of the assertions – how well they are supported by evidence, how well they advance thinking in the field – is more important than quantity. Few full-scale studies have more than five. • Where do the assertions come from? If you’ve developed categories and looked across them, your assertions may come from your categories and the linkages among them. The data elements in the categories are your support. Your memos may also provide seeds of assertions.
Pitfalls of Assertions • Under interpretation • Problem: Fails to make as strong a contribution to our understanding as could be made on the basis of the data. • Strategy: Write and rewrite statements that might be assertions, increasing the interpretive load each time, until you’ve gone as far as you the thing data will support. Then, seek critique.
Pitfalls of Assertions • Over interpretation • Overgeneralization - Common type of over interpretation. A claim that what was learned appears in a context that was not studied. • Problem: Claims we understood more than we do, more than the data supported. • Strategies: • Avoid statements that suggest applicability in unstudied contexts. • Be careful to qualify as much as possible – problems with access to people, items, time, etc. • Write down your assertions, and under each, list the data you have in support of it. Judge the sufficiency of the data that warrants your assertions. • Seek critique.
Assertions • Don’t have procedures to guarantee success, therefore, need to provide a logical line such as: observation interview interpretation. • Can use quotes from participants as assertions. • Often confused with assertions: • Observations • Generalizations* • Assumptions • Speculations • Common knowledge *An assertion can contain some degree of generalization, but should be petite generalization