IT 499: Seminar Course Week 3 Faculty: Dr. Afshan Jafri 28 February 2012

1. IT 499: Seminar CourseWeek 3Faculty: Dr. Afshan Jafri28 February 2012

2. Outline What is research? How to prepare yourself for IT research? How to identify and define a good IT research problem? Research Area Research Question / Topic How to solve it? Research methods Research phases How to write and publish an IT paper? Research Ethics 2

3. How to solve it? • Understanding the problem • Distinguishing the unknown, the data and the condition • Devising a plan • Connecting the data to the unknown, finding related problems, relying on previous findings • Carrying out the plan • Validating each step, (if possible) proving correctness • Looking back • Checking the results, contemplating alternative solutions, exploring further potential of result/method

4. Research Methods

5. A Research Method Classification • Scientific: understanding nature • Engineering: providing solutions • Empirical: data centric models • Analytical: theoretical formalism • Computing: hybrid of methods From W.R.Adrion, Research Methodology in Software Engineering, ACM SE Notes, Jan. 1993

6. Scientist vs. Engineer • A scientist sees a phenomenon and asks “why?” and proceeds to research the answer to the question. • An engineer sees a practical problem and wants to know “how” to solve it and “how” to implement that solution, or “how” to do it better if a solution exists. • A scientist builds in order to learn, but an engineer learns in order to build

7. The Scientific Method Observe Propose a model Measure and analyze Validate model/theory Repeat Observe real world Propose a model or theory of some real world phenomena Measure and analyze above Validate hypotheses of the model or theory If possible, repeat

8. The Engineering Method Observe (solutions) Propose improvement (build/develop) Measure and analyze Repeat, until no further improvement Observe existing solutions Propose better solutions Build or develop better solution Measure, analyze, and evaluate Repeat until no further improvements are possible

9. The Empirical Method Propose a model Develop statistical or other methods Apply to case studies Measure and analyze Validate Repeat Propose a model Develop statistical or other basis for the model Apply to case studies Measure and analyze Validate and then repeat

10. The Analytical Method Propose a formal theory or set of axioms Develop a theory Derive results Compare with empirical observations (if possible) Refine theory Propose a formal theory or set of axioms Develop a theory Derive results If possible, compare with empirical observations Refine theory if necessary

11. Computing

12. Research Phases

13. Research Phases • Informational: gathering information through reflection, literature, people survey • Propositional: Proposing/formulating a hypothesis, method, algorithm, theory or solution • Analytical: analyzing and exploring proposition, leading to formulation, principle or theory • Evaluative: evaluating the proposal R.L. Glass, A structure-based critique of contemporary computing research, Journal of Systems and Software Jan (1995)

14. Method-Phase Matrix

15. Example - Software Engineering • Informational phase - Gather or aggregate information via • reflection • literature survey • people/organization survey • case studies • Propositional phase - Propose and build hypothesis, method or algorithm, model, theory or solution • Analytical phase - Analyze and explore proposal leading to demonstration and/or formulation of principle or theory • Evaluation phase - Evaluate proposal or analytic findings by means of experimentation (controlled) or observation (uncontrolled, such as case study or protocol analysis) leading to a substantiated model, principle, or theory.

16. Computing Schism … • “ … computing research … is characterized largely by research that uses the analytical method and few of its alternatives … the evaluative phase is seldom included.” [Glass, 95] • “Relative to other sciences, the data shows that computer scientists validate a smaller percentage of their claims [Tichys 98]

17. SCIENTIFIC METHOD

18. Formulate Research Hypotheses Typical hypotheses: Hypothesis about user characteristics (tested with user studies or user-log analysis, e.g., clickthrough bias) Hypothesis about data characteristics (tested with fitting actual data) Hypothesis about methods (tested with experiments): Method A works (or doesn’t work) for task B under condition C by measure D (feasibility) Method A performs better than method A’ for task B under condition C by measure D (comparative) Introduce baselines naturally lead to hypotheses Carefully study existing literature to figure our where exactly you can make a new contribution (what do you want others to cite your work as?) The more specialized a hypothesis is, the more likely it’s new, but a narrow hypothesis has lower impact than a general one, so try to generalize as much as you can to increase impact But avoid over-generalize (must be supported by your experiments) Tuning hypotheses

19. Hypothesis Procedure Clearly define the hypothesis to be tested (include any necessary conditions) Design the right experiments to test it (experiments must match the hypothesis in all aspects) Carefully analyze results (seek for understanding and explanation rather than just description) Unless you’ve got a complete understanding of everything, always attempts to formulate a further hypothesis to achieve better understanding

20. Clearly Define a Hypothesis A clearly defined hypothesis helps you choose the right data and right measures Make sure to include any necessary conditions so that you don’t over claim Be clear about any justification for your hypothesis (testing a random hypothesis requires more data than testing a well-justified hypothesis)

21. Design the Right Experiments Flawed experiment design is a common cause of rejection of a paper (e.g., a poorly chosen baseline) The data should match the hypothesis A general claim like “method A is better than B” would need a variety of representative data sets to prove The measure should match the hypothesis Multiple measures are often needed (e.g., both precision and recall) The experiment procedure shouldn’t be biased Comparing A with B requires using identical procedure for both Common mistake: baseline method not tuned or not tuned seriously Test multiple hypotheses simultaneously if possible (for the sake of efficiency)

22. Carefully Analyze the Results Do the significance test if possible/meaningful Go beyond just getting a yes/no answer If positive: seek for evidence to support your original justification of the hypothesis. If negative: look into reasons to understand how your hypothesis should be modified In general, seek for explanations of everything! Get as much as possible out of the results of one experiment before jumping to run another Don’t throw away negative data Try to think of alternative ways of looking at data

23. Modify a Hypothesis Don’t stop at the current hypothesis; try to generate a modified hypothesis to further discover new knowledge If your hypothesis is supported, think about the possibility of further generalizing the hypothesis and test the new hypothesis If your hypothesis isn’t supported, think about how to narrow it down to some special cases to see if it can be supported in a weaker form

24. Derive New Hypotheses After you finish testing some hypotheses and reaching conclusions, try to see if you can derive interesting new hypotheses Your data may suggest an additional (sometimes unrelated) hypothesis; you get a by-product A new hypothesis can also logically follow a current hypothesis or help further support a current hypothesis New hypotheses may help find causes: If the cause is X, then H1 must be true, so we test H1

25. ENGINEERING METHOD

26. Research Cycle

27. Validation

28. EMPIRICAL METHODS

29. Components of Empirical Research • Problem statement, research questions, purposes, benefits • Theory, assumptions, background literature • Variables and hypotheses • Operational definitions and measurement • Research design and methodology • Instrumentation, Experiment • Data analysis • Conclusions, interpretations, recommendations

30. What is an Experiment? • Research method in which • conditions are controlled • so that 1 or more independent variables • can be manipulated to test a hypothesis • about a dependent variable • Allows • evaluation of causal relationships among variables • while all other variables are eliminated or controlled.

31. Variables • Dependent Variable • Criterion by which the results of the experiment are judged. • Variable that is expected to be dependent on the manipulation of the independent variable • Independent Variable • Any variable that can be manipulated, or altered, independently of any other variable • Hypothesized to be the causal influence

32. Setup • Experimental Treatments • Alternative manipulations of the independent variable being investigated • Experimental Group • Group of subjects exposed to the experimental treatment • Control Group • Group of subjects exposed to the control condition • Not exposed to the experimental treatment • Serve as the standard for comparison

33. Testing • Test Unit • Entity whose responses to experimental treatments are being observed or measured • Randomization • Assignment of subjects and treatments to groups is based on chance • Provides “control by chance” • Random assignment allows the assumption that the groups are identical with respect to all variables except the experimental treatment

34. Steps in Empirical Research Experiment Design(tweak software, establish experimental variables, procedure, design, run pilot subjects) Build Apparatus(integrate prototype and test conditions into experimental apparatus & software) User Study (collect data, conduct interviews) Analyse Data(build models, check for significant differences, etc.) Publish Results Next iteration

35. RECAP

36. Recap • What are the claims? • What are the factors and considerations? • What is the evaluation approach? • What are metrics? • How’s the data collected? • What are the results compared to? • How did you validate your results? • What are your findings?

37. The Claims • At the outset, there’s an indicated purpose to the work • Consequently, the purpose needs to be clearly defined • Also, the evaluation needs to be aligned to fit the purpose • So, what are your claims?

38. Types of Claims… • Reduced requirements • Better performance • Ease of use • Higher utility • Cost reduction • Best practice • Insights • Etc …

39. The Considerations • What are the inputs? • What factors affect your results? • Are they all variable? • Are they all controllable? • Is it possible to isolate their effect • Individually vs. collectively • Completely vs. partially?

40. Approach • Carry the thrust from clear purpose and clear understanding factors • Requires care as it will be subject to scrutiny • Must be structured and logical • Must exhibit an appropriate level of sophistication • Must be clear

41. Metrics • Define them • How were they measured before (if applicable)? • Are previous measures good enough?

42. Data Collection • Could be made through … • Measurements • Simulation • Survey • Who/what collected the data? • To what extent was the data collection process trustworthy? • How is this “extent” verified?

43. Baselines • Is there a need for baseline? • Can you establish a reasonable baseline? • Previous proposal(s) • Random behavior • Optimal behavior • Current behavior • Are you creating the baseline? (i.e., benchmarking?)

44. Validating Your Results • In part, stems from the validity of what’s been leading up to the results section • There’s also the breadth and depth of evaluation • Breadth: The span of cases and considerations under which the evaluation was performed • Depth: How far you go with each case or factor? • Scope of evaluation also raises credibility

45. Credits • http://www.cs.usyd.edu.au/~info5993/ • http://www.cs.uiuc.edu/homes/czhai • Abd Elhammid Taha, Research Methods, slides • From W.R.Adrion, Research Methodology in Software Engineering, ACM SE Notes, Jan. 1993 • Matti Tedre; “Teaching the Philosophy of Computer Science”, Journal of Information Technology Education; Know Your Discipline, 2007.