Johnny can’t Program (and neither can Johann)

Johnny can’t Program(and neither can Johann) Lynda A. Thomas Department of Computer Science, University of Wales, Aberystwyth, Wales, UK

Two Multi-institutional and Multi-national Studies of Beginning Programmers(and some attempts at solutions) • A multi-national, multi-institutional study of assessment of programming skills of first-year CS students Report by the ITiCSE 2001 Working Group Mike McCracken (et al.) SIGCSE Bulletin (Dec 2001) • A multi-national study of reading and tracing skills in novice programmers Report by the ITiCSE 2004 Working Group Raymond Lister (et al.) SIGCSE Bulletin (to appear) • Various papers by Lynda Thomas and Mark Ratcliffe

Outline • Outline the problem • Report on McCracken Study • Report on Lister Study • Some possible remedies • Discussion

Programming is hard! • du Boulay (1989) describes the sources of difficulty, and the need to deal with all of them at once: General orientation, notional machine, notation, structures, pragmatics • Soloway and Spohrer (1989) note deficits in understanding of concepts, shortcomings in planning and testing, and much more • Winslow (1996) notes that novices lack detailed mental models, are limited to surface knowledge and approach the problem line-by-line as opposed to in chunks

Does OO make it worse? • Rist (1996) suggests that is not so much that OO is different, it is more • Wiedenbeck (1999) compared OO and Procedural novices and discovered that although for short programs there was little difference in comprehension, programs with multiple classes were not understood as well. • However, in a later study it appeared that almost all difference was coming from weaker students.

On a Personal Note • 20 years of experience, US and UK • Old assignments - would I give them now? (note how more comes into this (e.g.. solitaire)) • There is a lot more to teach • The population is different Many of my students cannot program (30%?) Is it my fault?

McCracken Study • Participants from 8 universities, 5 countries were given a choice of 3 problems that students ‘should’ be able to solve (the problems were calculators). • Then gave the problem to an entire class of students at the appropriate level. • Solutions were marked by several researchers.

McCracken, et al. (2001) “Finger painters” • Most were end of first year students • Remember, 8 universities, 5 countries … • … it says something about our discipline. The “Picassos”

Why??? • According to McCracken et al, solving a programming problem involves the following … • (1) Abstract the problem from informal description, • (2) Generate sub-problems, • (3) Produce sub-solutions, • (4) Re-compose, and • (5) Evaluate and iterate

Why??? • According to McCracken et al, solving a programming problem involves the following … • (1) Abstract the problem from informal description, • (2) Generate sub-problems, • (3) Produce sub-solutions, • (4) Re-compose, and • (5) Evaluate and iterate “Oh, my students know how to code, but they can’t design.” (or “problem solve”)

Lister Study • The nature of the McCracken study does not isolate the exact problem • This study attempted to separate the steps so that we could see whether the problem was at the problem solving/design step or at the programming level • Students attempted 12 multiple choice questions. • Generic, 3GL, iterative processes on arrays • Could be translated into many languages, as it happens …Java (11 universities) and C++ (1 university) • 5 from US, 3 from Scandinavia, 3 from Australasia , 2 from UK

Multiple Choice Question 2, of 12 (a “fixed code” question) int[] x1 = {1, 2, 4, 7}; int[] x2 = {1, 2, 5, 7}; int i1 = x1.length-1; int i2 = x2.length-1; int count = 0; while ((i1 > 0 ) && (i2 > 0 )) { if ( x1[i1] == x2[i2] ) { ++count; --i1; --i2; } else if (x1[i1] < x2[i2]) --i2; else --i1; } After the while loop finishes, “count” contains what value? a) 3 b) 2 c) 1 d) 0 ie. 3 (Formatting compressed to fit question into this space.)

int[] x1 = {1, 2, 4, 7}; int[] x2 = {1, 2, 5, 7}; int i1 = x1.length-1; int i2 = x2.length-1; int count = 0; while ((i1 > 0 ) && (i2 > 0 )) { if ( x1[i1] == x2[i2] ) { ++count; --i1; --i2; } else if (x1[i1] < x2[i2]) --i2; else --i1; } ie. 3 Multiple Choice Question 2, of 12 (a “fixed code” question) After the while loop finishes, “count” contains what value? a) 3 b) 2 c) 1 d) 0 Wrong, 23% Correct, 65% (Formatting compressed to fit question into this space.)

Multiple Choice Question 8, of 12 (a “skeleton code” question) But first, the preamble to the question … If any two numbers in an array of integers, not necessarily consecutive numbers in the array, are out of order (i.e. the number that occurs first in the array is larger than the number that occurs second), then that is called an inversion. For example, consider an array “x” that contains the following six numbers: 4 5 6 2 1 3 There are 10 inversions in that array, as: x[0]=4 > x[3]=2 x[0]=4 > x[4]=1 x[0]=4 > x[5]=3 x[1]=5 > x[3]=2 x[1]=5 > x[4]=1 x[1]=5 > x[5]=3 x[2]=6 > x[3]=2 x[2]=6 > x[4]=1 x[2]=6 > x[5]=3 x[3]=2 > x[4]=1

Multiple Choice Question 8, of 12 (a “skeleton code” question) int inversionCount = 0; for ( int i=0 ; i<x.length-1 ; i++ ) { for ( xxxxxx ) { if ( x[i] > x[j] ) ++inversionCount; } } … the “xxxxxx” should be replaced by: a) ( int j=0 ; j<x.length ; j++ ) b) ( int j=0 ; j<x.length-1; j++ ) c) ( int j=i+1; j<x.length ; j++ ) d) ( int j=i+1; j<x.length-1; j++ )

Multiple Choice Question 8, of 12 (a “skeleton code” question) int inversionCount = 0; for ( int i=0 ; i<x.length-1 ; i++ ) { for ( xxxxxx ) { if ( x[i] > x[j] ) ++inversionCount; } } … the “xxxxxx” should be replaced by: a) ( int j=0 ; j<x.length ; j++ ) b) ( int j=0 ; j<x.length-1; j++ ) c) ( int j=i+1; j<x.length ; j++ ) d) ( int j=i+1; j<x.length-1; j++ ) Correct, ~50%

Multiple Choice Question 8, of 12 (a “skeleton code” question) int inversionCount = 0; for ( int i=0 ; i<x.length-1 ; i++ ) { for ( xxxxxx ) { if ( x[i] > x[j] ) ++inversionCount; } } … the “xxxxxx” should be replaced by: a) ( int j=0 ; j<x.length ; j++ ) b) ( int j=0 ; j<x.length-1; j++ ) c) ( int j=i+1; j<x.length ; j++ ) d) ( int j=i+1; j<x.length-1; j++ ) The idiomatic loop Correct, ~50%

Multiple Choice Question 8, of 12 (a “skeleton code” question) int inversionCount = 0; for ( int i=0 ; i<x.length-1 ; i++ ) { for ( xxxxxx ) { if ( x[i] > x[j] ) ++inversionCount; } } … the “xxxxxx” should be replaced by: a) ( int j=0 ; j<x.length ; j++ ) b) ( int j=0 ; j<x.length-1; j++ ) c) ( int j=i+1; j<x.length ; j++ ) d) ( int j=i+1; j<x.length-1; j++ ) The idiomatic loop Correct, ~50% Wrong, ~30%

Students across all institutions (N=556), percentage correct for each of the 12 MCQs. Q2, 6th hardest Q8, 3rd hardest

Moral: a difficult question at one place is difficult elsewhere

Quartiles on the 12 Multiple Choice Questions 12 universities!!! 7 countries!!! >500 students (If you failed more than 25% of class) The third quartile determines the “bottom passing” student.

McCracken et al., 2001 Lister et al., 2004

The Story So Far • Performance Data … and now …

The Story So Far “Doodles” • Performance Data … and now … • Doodles • Did students draw pictures and what sort

“Doodles”

Doodle Performance Percentage of correct answers when students (N=56) use a particular doodle type on any question (12).

Blank Doodles 3rd hardest, ~50% correct 6th hardest, 65% correct

Doodle Discussion • Students appear to lack good doodling strategies • Should we teach doodling (see later) • Assess doodling?

The Story So Far • Performance Data • Doodles … and now …

The Story So Far • Performance Data • Doodles … and now … • Interviews, “Think Out Loud” • Transcripts from 38 students

A transcript from Question 2 This time I'm going to look to see what the question wants first. So trying to find the value of count. So now I'm looking to see what each variable is set to and what's happening inside the program. I'm going to go ahead and write down that x1's length is 4 and x2's length is 4. And then e loop, 4 is greater than 0 and 4 is greater than 0. So inside the loop, the, checking to see if the 5th, checking, noticing that I didn't subtract 1 from each of i1 and i2. So changing i1 to 3 and i2 to 3. So when the 3rd spot, last spot of each array, they're equal, so it enters the first if loop so count is now equal to 1; i1 equals 2 and i2 equals 2. Now we're going through the loop again. 2 is greater than 0 and 2 is greater than 0 so we're inside the loop. Checking the first "if" are the second indexes equal to each other? no. So going to the else if and ‘is 4 less than 5?’, which is true so i2 is subtracted 1. I2 now equals 1 and we're going back to the while loop again. 2 is greater than 0 and 1 is greater than 0. So checking the first if ‘is 4 equal to 2?’, which is false so checking the second if. “Is 4 less than 2?’, which is false so doing the else statement which is subtract 1 from i1. so i1 now equals 1. So going through the while loop again. 1 is greater than 0 and 1 is greater than 0. So checking the first if "is 2 equal to 2?" which is true so increment count by 1. Count is now 2. Subtract 1 from i1 which becomes 0 and i2 which becomes 0. So now the while loop fails because 0 is not greater than 0. So it asks for the value of count which is 2.

Possible Remedies? • Other languages? • Force students to produce doodles? • Support for collaborative work? • Technology? • Integrated Development Environments (IDEs) For example, BlueJ (Kolling etc.), TogetherJ (Borland), …..

In particular we wanted: • support for scaffolding of key concepts • support for collaborative learning, and • a way of collecting information on how our students actually approach the design and coding process So, we decided to build our own

Four experiments at Aberystwyth and how they have affected the design of our IDE – Vortex • Learning Styles • Scaffolding with Visual Materials • Pair Programming • Collaborative Design

1. Learning Styles • Various approaches: left/right brain, Meyers Briggs, Kolb Learning styles, … • Felder-Silverman model of preferred learning style. • This identifies what is easy for the student. Felder believes that students need to improve other styles of learning too.

Active: try things out, work with others Reflective: think things through, work alone Sensing: concrete, practical, facts, procedures Intuitive: conceptual, innovative, theories and meanings Visual: pictures, diagrams, flow-charts Verbal: written or spoken explanations Sequential: incremental, orderly steps Global: holistic, learn in large leaps Felder believes that all undergrad education should be inductive (specific to general) not deductive

Active: try things out, work with others Reflective: think things through, work alone Sensing: concrete, practical, facts, procedures Intuitive: conceptual, innovative, theories and meanings Visual: pictures, diagrams, flow-charts Verbal: written or spoken explanations Sequential: incremental, orderly steps Global: holistic, learn in large leaps

Coursework mark Exam mark n Overall 63% 56 % 107 Active 61 % 52 % 58 Visual 61 % 54 % 82 Experimented and concluded… Some (small) evidence that Active and Visual (biggest groups) learners needed more support

So, what did that mean for Vortex? • First, all first year students are encouraged to reflect on their learning style and pointed to types of materials • In the design of Vortex, we wanted to: • Add visual material, and • Encourage active learning

2. Scaffolding with Visual Materials person1 person2 Person person1 = new Person(“Fred”, “Aberystwyth”); Person person2 = new Person(“Bill”, “Borth”); person2 = person1; Fred Bill Aberystwyth Borth

person1 person2 Person person1 = new Person(“Fred”, “Aberystwyth”); Person person2 = new Person(“Bill”, “Borth”); person2 = person1; person1.setAddress(“Llan”); ------------------ What is person2’s address? Fred Bill Aberystwyth Borth

What we wanted • To encourage our students to draw Object Diagrams • How? By providing them with partially completed diagrams as scaffolding. • Ultimately add this facility to Vortex?

Research Questions • Is drawing some kind of Object diagram correlated with success in solving multiple-choice tracing questions? • Does providing students with scaffolding in the form of partially completed Object diagrams help them correctly answer multiple-choice tracing questions? • Do students who have been provided with this scaffolding continue to use it in such multiple-choice questions?

Group n Follow-up Test Avg - Just on tracing questions Beginners who do not use diagrams 23 47% Beginners who do use diagrams 45 68% Is drawing some kind of Object diagram correlated with success in solving multiple-choice tracing questions? The answer is a (very) qualified yes (compare Lister)

Does providing students with scaffolding in the form of partially completed Object diagrams help them correctly answer multiple-choice tracing questions? We thought that the answer to this would be an obvious ‘yes’ but it appears to be ‘not really’ 36% for people given diagrams 28% for those not given diagrams

Do students who have been provided with this kind of scaffolding continue to use it? Not very surprisingly in light of last question, ‘no’

Why? In Hegarty and Narayanan model, the viewer: • decomposes the system into simpler components, • constructs a static model by making representational connections to prior knowledge and other components, • integrates information between different representations (e.g. text and diagrams), • hypothesizes lines of action, and finally • constructs a dynamic mental model by mental animation.

Why? In Hegarty and Narayanan model, the viewer: • decomposes the system into simpler components, • constructs a static model by making representational connections to prior knowledge and other components, • integrates information between different representations (e.g. text and diagrams), • hypothesizes lines of action, and finally • constructs a dynamic mental model by mental animation. We did They did?

Johnny can’t Program (and neither can Johann)