PART 2

1. PART 2 FOCUSING YOUR RESEARCH EFFORT (CHAPTER 5)

2. Successful research is a planned research Fail because plunged into activity with only a partially thought-out plan & inconclusive design Research requires a conceptualisation of overall organization and detailed plan Should be no less totally visualized and precisely detailed Planning the Research Project

3. A general strategy = research design (RD) RD provides Overall structure for the procedures The data the researcher collects The data analyses the researcher conducts Must plan the overall design carefully to be successful Wasted effort if half-prepared (vague ideas) Must identify RESOURCES, PROCEDURES, and DATA always with the goal from the very beginning Planning the Research Project

4. No matter what the academic discipline Not parochial (narrow, limited) and seldom fits into neatly packaged academic disciplines Software engineer may be compelled to stray far from the field of SE in pursuing a research problem in user interface design The ramifications of problem may lead into fields far from that of the original problem Err in thinking too narrowly, in restricting problems for research to one academic area All Research has a Basic Format

5. 1. A question is posed. In the mind of the researcher, a question arises that has no known resolution 2. It’s a matter of words. The researcher converts the question to a clearly stated research problem. 3. It’s worth a guess! The researcher poses a temporary hypothesis or series of hypotheses. 4. The search is on! The researcher searches the literature for ideas that shed light on the problem and for strategies that may help to address it. The Basic Format of the Research Process

6. 5. Data! Hard Data! And nothing BUT the data. The researcher collects data that potentially relate to the problem. 6. How do the data fit together? The researcher arranges the data into a logical organizational structure. 7. The data speak! The researcher analyzes and interprets the data to determine their meaning 8. It’s either … or … Either the data seemingly resolve the research problem or they do not. Either they support the hypotheses or they do not. The Basic Format of the Research Process

7. Might do better to think of problems as arising out of broad generic areas within whose boundaries all research falls: People – research problems relating to individuals, groups, populations, nationalities, families, ... Things – biological life, matter, … Records – letters, legal documents, lists, journals, … Thoughts and ideas – opinions, reactions, concepts, theories, … Dynamics and energy – human energy and activity, metabolism, gravity, hydrologic cycles, … Above suggestions are not all-inclusive, merely suggest the broad ramification of research possibilities Substratum (an underlying layer or foundation) of research principles is all-inclusive All Research has a Basic Format

8. Physicist exploring subatomic particles and sociologist exploring social behaviour are both employing the same research principles, although tools may differ, interpretational techniques may have little in common – approach identical Research approaches all problems through certain methodological channels appropriate to the nature and type of data Must differentiate research design VS research methodology All Research has a Basic Format

9. Many people are confused between the two Example, statement “research in physics is different from research in philosophy or history” Confusion between research as a process and the methodology employed by separate academic disciplines in collecting and processing data Genuine research follows the scientific method and exhibits the EIGHT characteristics Research Planning Versus Research Methodology

10. Data vary widely, therefore cannot employ the same methodology, e.g. blood cells vs historical facts In planning, the researcher must choose a viable research problem and consider the nature of data and the feasible means of collecting and interpreting those data Comparing the brain wave patterns of gifted and nongifted children may be an engaging project for research, BUT consider the following issues: Research Planning Versus Research Methodology

11. Who are willing to participate? Do we have the equipment? If so, do we have the technical skills to use it? Are we able to interpret the tracings? Do we know how to draw conclusion? It must be a practical research – something that we must be able to handle Research Planning Versus Research Methodology

12. Universality – Could be carried out by any competent person other than the researcher, same results without prejudice to project and validity Replication – Should be repeatable by other researchers under the same circumstances, within identical parameters and achieve comparable results Control – All researches are conducted within an area sealed off by given parametric limitations General Criteria for a Research Project

13. Measurement – Data should be susceptible to measurement. Easily accomplished in the physical sciences. In humanities and social sciences research, it is more difficult to quantify, measure, or evaluate Many research efforts are wasted by going-off HALF-PREPARED Must bring all together, early inventory of resources, your problems, and the source of data, in order to save time, money and effort Criteria for Research Project

14. Without facts (data), inductive reasoning vanishes and scientific method collapses The lifeblood of research Data is a Latin verb dare, meaning “to give” – give information, impression or other factual data to an observer Facts from Latin word facere, meaning “to make” – what the situation makes or manifests to the observer Data: Their Relationship to Design and Methodology

15. Data are manifestation of the truth rather than the truth itself – no one has ever look upon truth itself – pure, undisguised, naked Truth Data are merely representative, intermediate, elusive surrogate of Truth – reflects Truth as a mirror which reflects sunlight Researchers are like those who live in dungeon seeing a beam of sunlight passes across the floor and get the idea of what the sun must be like What is Data?

16. Factual dungeon – never be able to see the original source of the data – see what other people do – the behaviours they exhibit, the things they create, and the effects of their actions on others. But the actual people “inside” – those individuals we shall never know! Research seeks, through data, what is true absolutely – the complete meaning of the data Experienced researchers are constantly aware that the ultimate goal of research (Truth) is forever just beyond what is represented by the data, and, hence, just beyond human grasp What is Data?

17. The relationship between data and the ultimate truth is shown in the diagram - various states of data Realm of ultimate truth, the most inaccessible – can be approached by means of passing through two intermediate areas labeled The Realm of the Data What is Data?

18. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? The Realm of the Inquisitive Mind of the Researcher o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o o The Barrier of the Human Senses Skills of Reading and Writing, Language Capability, and the Channels of Communication The Region of the Secondary Data The Realm of the Data The Dividing Line Between Primary and Secondary Data      The Region of the Primary Data The Impenetrable Barrier Beyond Which Lies the Absolute Truth THE REALM OF ABSOLUTE TRUTH

19. Data are not only elusive but also transient – exist in a split second Example, an opinion survey on a population – tomorrow’s result will not be the same as now, people move out, move in, died, …. Pretest and posttest conducted by a teacher in a class may not be the same class at all in six weeks, they have been growing, learning, maturing, changing Data are Transient and Ever Changing

20. Example, asking people about their interests – can we be sure what they tell us is true? They may answer honestly that represents the truth as they perceived it at that instant May change because of factors: age, maturity, newscast, or items in newspaper Researcher should recognize that even the most reliable, refined, carefully controlled data may have elusive (hard to find, describe or remember) quality – very volatile, evaporate Data are Transient and Ever Changing

21. Lie closest to the source of The Absolute Truth underlying the phenomenon Reflect truth more faithfully than any other approach – most valid, most illuminating, the most truth-manifesting Impenetrable barrier exists between the Realm of Absolute Truth and the region of primary data A beam of sunlight does not compare with seeing the sun itself Primary Data

22. I witnessed the full occurrence of a car veer off the highway and careen into the ditch The driver told me that he had no awareness of the possibility of an accident Neither of us will ever be able to determine the absolute truth underlying the accident What are the factors? Momentary seizure? Mechanism imperfections? Answers lie beyond an impenetrable barrier, may never be known Car Accident

23. Beyond the region of primary data lies the region of secondary data In the dungeon – not a direct beam of sunlight but simmering light upon the wall, reflected by mirror, distorted by imperfections In the case of the car accident, I wrote an account stating precisely what I observed and confirmed by the driver – readers of the next day’s newspaper get the similar “reflection of the sunlight” Secondary Data

24. The data/facts are of necessity distorted albeit by the channels of communication, my writing skills, reading skills, and the inability of language to produce every detail Many more barriers than in the diagram The acuity of human senses The sensitivity of instruments The failure of language to communicate thought exactly The inability of two persons to witness the same event and report it precisely as duplicate accounts Barriers of The Truth

25. Even in exuberant moment of discovery, no one has ever glimpsed the Absolute Truth, nor can we come to a knowledge of the data that reflect that Truth except through the gross and shadowy channels of our dull and imperfect senses Be cautious and respect for words such as perhaps, it seems, one might conclude and it would appear to be Unwarranted Conclusion

26. Not all data are acceptable – can be defective – affect the validity of the conclusion Stem from imperfections and irregularities of nature Research should be replicable under precisely the same conditions To regulate the precision of conditions, criteria must be adopted, limits established, standards set up that all data must meet Criteria for Admissibility of Data

27. Non-conformance excluded These controls are easy in sciences Standardize data, admit only compliance – can control the research effort and conclude with greater certainty The criteria must be set forth clearly in proposal as well as report to ensure integrity of research So the consumer can come to an intelligentunderstanding of what is being studied Criteria for Admissibility of Data

28. Sometimes we will be able to use one or more existing measurement instruments OR have to develop our own instruments Measurement instruments provide a basis on which the entire research effort rests Describe any instrument used in explicit, concrete terms Provide evidence that the instruments you use have a reasonable degree of validity and reliability for your purposes Validity = the extent to which the instrument measures what it is actually intended to measure Reliability = the extent to which it yields consistent results when the characteristic being measured hasn’t changed Identifying Appropriate Measurement Instruments

29. Face validity The extent to which, on the surface, an instrument looks like it is measuring a particular characteristic Useful for ensuring the cooperation of people who are participating in a research study Since it relies entirely on subjective judgment, it is not, in and of itself, terribly convincing evidence that an instrument is truly measuring what the researcher wants to measure Determining the Validity of Measurement Instruments

30. Content validity The extent to which a measurement instrument is a representative sample of the content area (domain) being measured Used to assess people’s achievement in some area – for instance, the knowledge they have learned during classroom instruction or the job skills they have acquired in a rehabilitation program High content validity means that its items or questions reflect the various parts of the content domain in appropriate proportions and requires the particular behaviours and skills that are central to that domain Determining the Validity of Measurement Instruments

31. Criterion validity The extent to which the results of an assessment instrument correlate with another, presumably related measure (the criterion) Example, a personality test designed to assess a person’s shyness or outgoingness (introvert or extrovert) has criterion validity if its scores correlate with other tests of introversion versus extroversion An instrument designed to measure a salesperson’s effectiveness on the job should correlate with the number of sales the individual actually makes during the course of a business day Determining the Validity of Measurement Instruments

32. Construct validity The extent to which an instrument measures a characteristic that cannot be directly observed but must instead be inferred from patterns in people’s behaviour (such a characteristic is a construct) Motivation, creativity, racial bias, bedside manner – all of these are constructs, in that none of them can be directly observed and measured Sometimes there is universal agreement that a particular instrument provides a valid instrument for measuring a particular characteristic We could all agree that a ruler measures length, a thermometer measures temperature, and a barometer measures air pressure But whenever we do not have such universal agreement, we must provide evidence that an instrument we are using has validity for our purpose Determining the Validity of Measurement Instruments

33. To demonstrate that the measurement instruments used by researchers have validity for their purposes, the following examples are used: - A multitrait-multimethod approach Two or more different characteristics are each measured using two or more different approaches. The different measures of the same characteristic should be highly correlated. The same ways of measuring different characteristics should not be highly correlated - A table of specifications To construct a measurement instrument that provides a representative sample of a particular content domain – in other words, to establish content validity – the researcher often constructs a two-dimensional grid (table of specifications) listing the specific topics and behaviours that reflect achievement in the domain. In each cell of the grid, the researcher indicates the relative importance of each topic-behaviour combination. He or she then develops a series of tasks or test items that reflects the various topics and behaviours in appropriate proportions - Judgment by a panel of experts Several experts in a particular area are asked to scrutinize an instrument to ascertain its validity for measuring the characteristic in question Although none of the approaches just described guarantees the validity of a measurement instrument, each one increases the likelihood of such validity. Determining the Validity of Measurement Instruments

34. Interrater reliability – the extent to which two or more individuals evaluating the same product or performance give identical judgments Internal consistency reliability – the extent to which all the items within a single instrument yield similar results Equivalent forms reliability – the extent to which two different versions of the same instrument (e.g., “Form A” and “Form B” of a scholastic aptitude test) yield similar results Test-retest reliability – the extent to which the same instrument yields the same result on two different occasions Determining the Reliability of Measurement Instruments

35. A researcher can enhance the reliability of a measurement instrument in several ways. 1) The instrument should always be administered in a consistent fashion; in other words, there should be standardisation in use of the instrument from one situation or person to the next 2) To the extent that subjective judgments are required, specific criteria should be established that dictate the kinds of judgments the researcher makes 3) Any research assistants who are using the instrument should be well trained so that they obtain similar results Reliability and Validity

36. We can measure something accurately only when we can also measure it consistently In other words, in order to have validity, we must also have reliability The more valid and reliable our measurement instruments are, the more likely we are to draw appropriate conclusions from the data we collect and, thus, to solve our research problem in a credible fashion Reliability and Validity

37. Data are like ore – contain pieces of the truth but are in a rather unrefined state To extract meaning from the data, we employ what is commonly called research methodology (RM) Data and methodology are inextricably interdependent The RM to be used for a particular research problem must always take into account the nature of the data that will be collected in the resolution of the problem RM is merely an operational framework within which the facts are placed so that their meaning may be seen more clearly Linking Data and Research Methodology

38. Different questions yield different types of information Different research problems lead to different research designs and methods, which in turn result in the collection of different types of data and different interpretations of those data Many kinds of data may be suitable only for a particular methodology Data and RM

39. To some extent, the data dictate the research method Historical data from written records of past event - cannot extract meaning by laboratory experiment. An experiment is simply not suited to the nature of the data Not only a true experiment constitutes a “research” Data and RM

40. Two types, writings and observations Written records and accounts – of past happenings Observation for whose transmission description is the best vehicle - at the scene of occurrence Observation that are quantified and exist in the form of numerical concepts – in the language of mathematics Observation of certain differences and likeness that arise from comparison or contrast of one set of observations with another set of similar observations Nature of Data

41. Four kinds of data demand four discrete and different research methodologies The data dictate the RM Unless fits a given methodology, it fails to be a research Research Methodology (RM)

42. 1-1 correspondence between data and RM Historical method – for documentary data or literary Descriptive survey method (normative survey method) – data from observational situation – physically observed or through questionnaire or poll techniques Analytical survey method – quantitative data that need statisticsto extract their meaning Experimental method – appropriate for data derived fromexperimental control or pretest-posttest design, two separate groups or onegroup from which data are derived at two separate intervals Research Methodology (RM)

43. Quantitative research– is used to answer questions about relationships among measured variables with the purpose of explaining, predicting, and controlling phenomena. This approach is sometimes called the traditional, experimental, or positivist approach Qualitative research– is used to answer questions about the complex nature of phenomena, often with the purpose of describing and understanding the phenomena from the participants’ point of view. This approach is also referred to as the interpretative, constructivist, or postpositivist approach Quantitative vs. Qualitative Research

44. Both approaches involve similar processes (e.g., formation of one or more hypotheses, review of the related literature, collection and analysis of data). Yet these processes are often combined and carried out in different ways, leading to distinctly different research methods Quantitative researchers usually start with a specific hypothesis to be tested, isolate the variables they want to study, control for extraneous variables, use a standardized procedure to collect some form of numerical data, and use statistical procedures to analyse and draw conclusions from the data Quantitative vs. Qualitative Research

45. Qualitative researchersoften start with general research questions rather than specific hypotheses, collect an extensive amount of verbal data from a small number of participants, organise those data into some form that gives them coherence, and use verbal descriptions to portray the situation they have studied Quantitative vs. Qualitative Research

46. A quantitative study usually ends with confirmation or disconfirmation of the hypotheses that were tested A qualitative study is more likely to end with tentative answers or hypotheses about what was observed. These tentative hypotheses may form the basis of future studies (perhaps quantitative in nature) designed to test the proposed hypotheses In this way,qualitative and quantitative approaches represent complementary components of the research process – appropriate for answering different kinds of questions As a result, we learn more about the world when we have both quantitative and qualitative methodologies Quantitative vs. Qualitative Research

47. Summary of Qualitative vs. Quantitative Approaches Please refer to Table 5.1 on page 96 of the textbook Deciding whether to use a quantitative or qualitative approach Please refer to Table 5.2 on page 106 of the textbook Quantitative vs. Qualitative Research

48. Action Research Case and Field Study Research Correlational Research Developmental Research Ex Post Facto or Causal-Comparative Quasi-Experimental Research Other RMs

49. Please refer to Table 5.3 on page 108 of the textbook Other RMs