Understanding Difficulty & complexity in writing tasks: Toward a useful analytic framework

1. TBLT 2009 Lancaster Understanding Difficulty & complexity in writing tasks: Toward a useful analytic framework Anne Nebel Lancaster University a.nebel@lancaster.ac.uk Georgetown University aln27@georgetown.edu School of Foreign Service in Qatar

2. TBLT research • Classroom context (sequencing tasks for learning) • Speaking tasks • Assessment context (Chalhoub-Deville, 2001; Elder, Iwashita & McNamara, 2002; Wigglesworth, 2001) • Writing tasks(Ishikawa, 2008; Kuiken & Vedder, 2007) • Predominant models developed for speaking (Robinson, 2001, 2005; Skehan & Foster, 1999, 2001)

3. Writing assessment tasks • My research investigates the difficulty and complexity of a set of 8 writing tasks • Not experimentally designed tasks • Fixed set of retired test tasks (ECCE) • Writing task option pair: letter or essay

4. Task choice? • Challenge of task equivalence • Within each test and across versions • Potential for eliciting sufficient output equally rich in features under assessment • Raises validity & reliability issues • Fairness is a primary concern • What decisions and conclusions are reached from the test results? (Bachman & Palmer, forthcoming)

5. Research context (1) • High-stakes nature of language proficiency assessment (e.g. Greece) • Employment opportunities • Career advancement • Pay scale • May impact performance (affective variables) • Over-trained and test-savvy • Test context as a facet in the complexity equation?

6. Research context (2) • Low-stakes environments (e.g. Qatar) • Opportunity to look at test tasks outside the test context • Test-like conditions (timed, no resources, etc.) • None of the affective and pragmatic interference of a real test situation • Motivation?

7. My research: writing assessment tasks • Perceptions of task difficulty • Student (N=20 pilot questionnaire and raking activity) • Teacher (N=5, 30-minute interviews & ranking activity) • Student (N=? interviews and raking activity) • Analysis of task complexity • 8 task set of 4 pairs (letter & essay) • Various model adaptations – toward the development of a framework • Analysis of performances on tasks • Measures of CAF (which, why) • Interactions between perceptions, task complexity and performance (what, how)

8. How to analyze difficulty & complexity? DIFFICULTY learner factors / perceptions COMPLEXITY task structure / analysis • Expert judgments, VPAs, interviews, surveys and questionnaires. (e.g. Elder, Iwashita and McNamara, 2002; Hamp-Lyons, 1994) • Existing models (Robinson, 2001, 2005; Skehan & Foster, 1999, 2001; and Ishikawa, 2008; Kuiken & Vedder, 2007) • Taxonomies of characteristics (Bachman & Palmer, 1990; Carr, 2006) • Coding & classification systems (Silver et al., 2009)

9. Student perceptions of difficulty (phase I) Task choice? letter (18), essay (2) S1: “I chose the letter because it gives me the way how to write and explain my opinion.” S4: “Letter, because it’s easiest.” S14: “Because I can explain a situation better in a letter than in the essay.” S17: “It’s easier to give an opinion on a letter (less formal) than writing an essay.” S18: “In my view a letter is more personal and it is easy to write their own opinion.” S6*: “Because I don’t know how to write an essay.”

10. “View from the learner”(Ortega, 2005) • “Whether through discourse analysis, psychological questionnaires, or post-task interviews, a process-product approach that considers the full landscape of variables contributed by task, leaner and linguistics outcomes, has the potential to illuminate our theories of the role of planning in task-based language learning” (Ortega, 2005: 116). • …of task complexity?

11. Approach to perceptions data (phase II) • Phenomenography Qualitative approach that makes use of in-depth interview data to explore how participants conceive of a phenomenon (Marton, 1981); • Reveals “categories of understanding” that can be ordered in an outcome space (Micari et al., 2007); • Characterizes “ways of seeing something in terms of the critical aspects of the phenomenon” (Pang, 2003: 145)

12. Theory and method • PCP (Kelly, 1955) / Repertory Grid Technique Personal construct theory is a theoretical framework that “provides a vocabulary with which to interpret the constituents and processes by which a person construes his or her experience” (Roberts, 1999:119). • Dyadic difference elicitation technique Elicit constructs based on a comparison of elements. “There is nothing sacrosanct about the triad” (Fransella, Bell and Bannister, 2004: 28).

13. Elements: 8 tasksConstruct elicited (sample): • Specific / non-specific • Personal / general (“private/public orientation” Hamp-Lyons & Mathias,1994: 54-55) • Real life / artificial or hypothetical • Meaningful / meaningless • Concrete / general or vague • Specified content / need to invent content • Voc-ed domain / academic domain • 1st person / 3rd person • Suggested structure / no suggestion of structure

14. Task ranking • Teachers (N=5) ranked 8 tasks in order of difficulty. • In every case, task 6 was ranked as the most difficult or the second most difficult. • In every case (but one), task 1 was ranked as the easiest or the second easiest task of the set.

15. Task analysis Task 6 (most difficult) Task 1 (easiest) (essay option, task pair 3) Should public and private resources be used to support development in locations where natural disasters, such as floods, forest fires, and volcano eruptions, could occur? Explain your views. Give specific reasons and examples. (http://www.lsa.umich.edu/UMICH/eli/Home/Test%20Programs/ECCE/General%20Information/2000ecceformb.pdf) (letter option, task pair 1) Write a letter to the Editor of the City Times. Name one person you think should go on the Millennium Exchange Program. Describe the person, and tell why you think this person should go. You may nominate yourself or someone else you know. (http://www.lsa.umich.edu/UMICH/eli/Home/Test%20Programs/ECCE/General%20Information/ECCE%20Practice%20Tests/ECCE_06NDWriting3.pdf)

16. Task complexity analysis: constructs Task 6 (rebuild city with funds essay) Task 1 (Millennium nominee letter) -specific -personal -concrete (abstract) -meaningful 6 = more complex task + specific + personal + concrete + meaningful

17. Task complexity analysis: (Silver et al., 2009) Task 6 (rebuild city with funds essay) Task 1 (Millennium nominee letter) • High cognitive demand • analyze • evaluate • interpret • create • justify • make decisions • 6=more complex task • Low cognitive demand • recall • remember • describe • guided • routine • constrained

18. Task complexity analysis: LACM and TCF(compared in Kuiken & Vedder, 2007) LACM (Skehan) TCF (Robinson) • Code complexity • Vocab • Cognitive complexity • Familiarity of topic • Predictability • Information organization 6=more complex task • Resource directing • Few elements • Here and now • No reasoning demands • Resource dispersing • Single task • Prior knowledge • Task difficulty

19. Task complexity analysis: 1 adaptationfor speaking (Elder, Iwashita and McNamara, 2002) 2 performance conditions (+/-) 4 task dimensions: • Perspective: your own/another’s • Immediacy: there-then/here-now • Adequacy: complete/incompleteinput • Planning time: [constant] Which is more complex: 6? Equal? Inconclusive

20. Complex tasks, complex writing? • Although they don’t always take into account the same task dimensions, the models are capable of accounting for complexity in writing tasks; which dimensions are essential and how do we operationalize them? • Can the models predict the corresponding features of writing that will demonstrate complexity of thinking in writing? Which features and why? • In language testing, an equally important consideration is how well we can measure these features with a scoring rubric or scale.

21. References Bachman, L. and Palmer, A. (1990) Language Testing in Practice. Oxford: Oxford University Press. Carr, N. (2006) The factor structure of test task characteristics and examinee performance. Language Testing 23, 269-289. Chalhoub-Deville, M. (2001) Task –based assessments: characteristics and validity evidence. In Bygate, M. Skehan, P. and Swain, M. (2001) Researching pedagogic tasks: second language learning, teaching and testing. Essex: Pearson Education Limited. Ch 10, pp. 210-228. Elder, K, Iwashita, N. and McNamara, T. (2002) Estimating difficulty or oral proficiency tasks: what does the test-taker have to offer? Language Testing 19 (4); 347-368. Fransella, F., Bell, R. and Bannister, D. (2004) A Manual for Repertory Grid Technique. West Sussex: John Wiley and Sons. Hamp-Lyons, L. and Mathias, S. (1994) Examining expert judgments of task difficulty on essay tests. Journal of Second Language Writing 3 (1); 49-68. Ishikawa, T. (2008) The effect of manipulating task complexity along the [+/- here and now] dimensions on L2 written narrative discourse. In M. Garcia May (ed) Investigating tasks in formal language learning. Clevedon, UK: Multilingual Matters. Kelly, G. (1955) The psychology of personal constructs. New York: Norton. Kuiken, F. and Vedder, I. (2007) Task complexity and measures of linguistics performance in L2 writing. IRAL 45, 261-284. Micari, M., Light, G., Calkins, S. and Streetweiser (2007) Assessment beyond performance: phenomenography in educational evaluation. American Journal of Evaluation 28 (40); 458-476. Marton, F. (1981) Phenomenography: describing conceptions of the world around us [online version]. Instructional Science 10, 177-200. Retrieved 2/12/08 from http://www.ped.gu.se/bjorn/phgraph/misc/constr/html)

22. References Ortega, L. (2005) What do learners plan? Leaner-driven attention to form during pre-task planning. In Ellis, R. (ed) Planning and task performance in a second language. Amsterdam/Philadelphia: John Benjamins. Pp. 77-110. Pang, M. F. (2003) Two faces of variation—on continuity in the phenomenographic movement. Scandinavian Journal of Educational Research 47 (2); 14-156. Roberts, J. (1999) PCP as a framework for research into teacher learning and thinking. Language Teaching Research 3 (2); 117-144. Robinson, P. (2001) Task complexity, task difficulty and task production: exploring interactions in a componential framework. Applied Linguistics 22, 27-57. Robinson, P. (2005) Cognitive complexity and task sequencing: studies in a componential framework for second language task design. IRAL 43, 1-32. Silver, E., Mesa, W., Morris, K., Star, J. and Benken, B. (2009) Teaching mathematics for understanding: an analysis of lessons submitted by teachers seeking NBPTS certification. American Educational Research Journal, 46 (2); 501-535. Skehan and Foster (1999) The influence of task structure and processing conditions on narrative retellings. Language Learning 49, 93-100. Skehan and Foster (2001) Cognition and tasks. In P. Robinson (ed) Cognition in second language instruction (pp. 185-205). Cambridge: Cambridge University Press. Wigglesworth, g. (2001) Influences on performance in task-based oral assessments. In Bygate, M. Skehan, P. and Swain, M. (2001) Researching pedagogic tasks: second language learning, teaching and testing. Essex: Pearson Education Limited. Ch 9, pp. 186-209.

23. Thank you. Anne Nebel a.nebel@lancaster.ac.uk aln27@georgetown.edu