Affective Assessment of Team Skills in Agile CS1 Labs: The Good, the Bad, and the Ugly

Affective Assessment of Team Skills in Agile CS1 Labs:The Good, the Bad, and the Ugly Dawn McKinney and Leo F. Denton http://www.cis.usouthal.edu/~mckinney/SIGCSE2005.ppt School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

University of South Alabama • State university • 13,000 students • Computer and Information Sciences • 500 CIS majors • Three specializations • Undergraduate and Masters Program School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

OUR CS1 ENVIRONMENT • 2-semester course for majors • 75 minutes per week closed lab • Java programming language • Graduate assistants for support • Required personal laptops • Supplemental Instruction • Small class sizes School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

Bloom’s cognitive taxonomy Evaluate solutions to complex problems that involve class hierarchies and the use of polymorphism. Develop software that involves the use of new classes based on refining existing classes. Perform arithmetic operations on various data types. Explain source code and how it is produced. Define three kinds of program errors. School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

Krathwohl’s affective taxonomy Discuss specific personal initiatives which demonstrate a commitment to life-long learning. Show strong work ethic and initiative while working in groups. Use coding standards as taught in the lectures. Become aware that testing is a part of the software development life cycle. School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

NACE Top-10 characteristics most wanted in college graduates • Communication skills • Honesty/integrity • Teamwork skills • Interpersonal skills • Motivation/initiative • Strong work ethic • Analytical skills • Flexibility/adaptability • Computer skills • Organizational skills School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

Our studySpring 2004 • Team skills promoted in a closed lab • Integrated cognitive-affective objectives • Agile practices of Extreme Programming • Semester-long project • Quantitative and Qualitative Assessment • Improvements in Fall 2004 School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

Integrated cognitive-affective objectives for team skills School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

Forming of teams • Students formed their own teams • Between 5 and 9 students • Team names • Problems with unbalanced skill levels School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

Semester-long project • Teams were assigned the same project • Only during closed lab • “Real world” problem to solve • Three project iterations • Peer evaluations School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

Planning game Small releases Metaphor Simple design Test-driven development Refactoring Collective ownership Continuous integration Sustainable pace On-site customer Coding standards Stand-up meetings Agile Practices School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

High impact practices • Pair-programming • Collective code ownership • Test-driven development • On-site customer • Stand-up meetings School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

Pair-programming • Code in pairs only • Record pair programming experiences • Take turns at “driver” and “co-pilot” • “Pair pressure” • Cooperation • Communication of ideas • Develop and practice adaptability skills School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

Collective code ownership • Code kept on a server • Any team member could change the code • Adaptability • Communication • Cooperation School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

On-site customer • Customer made various project changes • Instructor was the customer • Adaptability • Cooperation • Normalcy of change School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

Test-driven development • Automated tool • New to students • Inexperienced programmers • Needed more instruction on TDD • Needed a more appropriate IDE • Adaptability • Cooperation School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

Stand-up meetings • 2 - 5 minute meetings • Each team member reports • Communication • Accountability • Cooperation • Commitment • Work ethic School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

Assessment instruments • Peer Evaluation • Final Peer Summary Evaluation • Intrinsic Motivation Inventory (IMI) • Anderson-Butcher’s Belonging Scale • Observation School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

First and second peer evaluations • Specific positive contributions of this team member • Best qualities this member added to the team effort • Weaknesses of this member • Would you want this member on your team again? (yes/no) • Overall level of contribution of this team member (Low to High on a 5 point Likert scale) School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

Frequency-of-use score as an unobtrusive measure of the internalization of team skills • Review of student evaluations • Identification of team-based skills • Assignment of frequency-of-use score • Correlation of the frequency of use score with: • Course grade • First and second overall peer evaluation scores • Final summary scores for each of the five skills School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

Measuring affective factors • Affective factors correlate with achievement • In past semesters we found a decrease in factors • Agile labs show increases in sense of belonging and lack of pressure School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

Significant positive correlations School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

Increase in belonging, decrease in pressure School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

Sample peer evaluations data School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

To round up the bad… • Provide timely feedback on peer evaluations • Supply instruction on team skills • Choose balanced and effective teams “He is set in his way of thinking” School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

To corral the ugly… • Deal with disrespectful or uncooperative students • Require problem students to work alone or among themselves “Maybe you should discuss appropriate behavior toward women with him.” School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

To carry on the good… • Promote team skills • Provide a semester-long lab project • Use agile practices • Conduct peer evaluations “She has an open mind and wants to hear and learn about what others bring to the table ” School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

Benefits of early team experiences • Assessment of affective objectives • Preparation for upper-level courses • Development of skills needed by industry • Natural active and cooperative learning • Women and other minorities • Sense of belonging School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

http://www.cis.usouthal.edu/~mckinney/SIGCSE2005.pptLeo F. Denton Dawn McKinneyldenton@usouthal.edudmckinney@usouthal.edu School of Computer and Information Sciences University of South Alabama, Mobile, Alabama USA

Affective Assessment of Team Skills in Agile CS1 Labs: The Good, the Bad, and the Ugly