The AERO CRCD “Smart Materials” Project 1 Preparation for Evaluation

The AERO CRCD “Smart Materials” Project 1 Preparation for Evaluation Adaptation, Evolution and Application of The TAMU-TIDEE Design Knowledge Test Rita Caso, Ripla Kohli, Jong-Hwan Lee, Texas A&M University Developed from Work In Progress presentation made at FIE Conference, Session 1199, 6-9 November , Boston, MA.

Acknowledgements This material is based upon work supported by the National Science Foundation under the following grants: EEC-9802942 EEC-0088118

Adaptation of the Transferable Integrated Design Engineering Education (TIDEE), Mid Program Assessment: A Three Part Assessment of Team Based Design for Entering Juniors 2, Part 1, and its evolution as the TAMU-TIDEE Design Knowledge Test Part I

Instrument Version Informed Judgment to Change Instrument or Rubric Instrument Scored with Indicated Program Rubric Baseline Data Collected for Program Evaluation Data Collected for Design Assessment Instrument Development Legend for..Diagrams of Instrument & Rubric Evolution, and Data Collection

Instrument & Rubric Evolution TIDEE Instrument Prototype TIDEE Rubric Prototype TAMU-TIDEE Test V.1 V.1 TAMU-TIDEE Rubric TAMU-TIDEE Test V.2 V.2 TAMU-TIDEE Rubric TAMU-TIDEE Test V.3 V.3 TAMU-TIDEE Rubric Future TAMU-TIDEE Test V.4

Data Collection TIDEE Instrument Prototype TIDEE Rubric Prototype Fall 2001 V.1 TAMU-TIDEE Test Freshmen Aero Seniors V.1 TAMU-TIDEE Rubric Spring 2002 V.2 TAMU-TIDEE Test Freshmen Aero Seniors Professors V.2 TAMU-TIDEE Rubric Fall 2002 V.3 TAMU-TIDEE Test EE Seniors* V.3 TAMU-TIDEE Rubric V.4 TAMU-TIDEE Test

TIDEE Definitions for Levels of Achievement in Design Education Outcomes Level Description Basic Knowledge Recognition and understanding of facts, terms, definitions, descriptions, relationships, processes. Application of Use of knowledge in ways that Knowledge demonstrate understanding of concepts or terms, their proper use, and limitations of their applicability Critical Analysis Examination and evaluation of informative required to judge its value to a design solution and to make decision. Extension of Extensive knowledge beyond what was received, creating new knowledge, making inferences, transferring knowledge to usefulness in new area of application, making improvements.

Key to ChangesMade in Next Version of Instrument • Text in red was identified as needing change in NEXT version • Example:Respond to the following questions/statements. You have 15 mins. • Text in red, set apart by “* *” indicates that something additional is needed in NEXT version • Example:*( Lacks specific instructions for response) * • Text in italics indicate new wording to that version • Example:In the space provided, describe your understanding of the engineering design process, in diagram or flowchart form, in complete sentences, or in list form. • Text which isboth italicised and in red indicates that the wording is new to that version, but that it was identified as needing change in NEXT version • Example: List the qualities that describe effective teamwork

TAMU AERO CRCD/TIDEE Prototype: The TIDEE Design Knowledge Assessment Instrument 3(TIDEE Website, June 2001) • Objective: Demonstrate your knowledge of key elements in the engineering design process, teamwork, and communication associated with team-based engineering design. • Assignment: Respond to the following questions/statements.You have 15 mins. • In a general sense, a process is an ordered set of activities to accomplish a goal. In the space provided, describe and/or diagram your understanding of the engineering design process. *( Lacks specific instructions for response) * • List the elements required for effective teamwork in a design project assignment, and explain why each element is important. • In team-based design, documentation and exchange of design information are important. List features of effective communication. Consider a wide range of oral and written communicationsituations. Davis, D. C. (2001). Transferable Integrated Design Engineering Education (TIDEE), Mid Program Assessment: A Three Part Assessment of Team Based Design for Entering Juniors. http://www/cea/wsu.edu/TIDEE

V.1--TAMU TIDEE Test: 1st Adaptation Given, as Pre-test, to Aero Senior Design Class and “Smart Material” Freshman Class in Fall 2001 • Objective: Demonstrate your knowledge of key elements in the engineering design process, teamwork, and communication associated with team-based engineering design. • Assignment: Respond to the following questions/statements as instructed(Time allowed 20-30 mins.) • In a general sense, a process is an ordered set of activities to accomplish a goal. In the space provided, describe and/or diagram your understanding of the engineering process. (Express yourself in complete sentences or in acombination of diagrams and clear phases.) • What kinds of things should teams do to prepare themselves beforehand for effective teamwork in a design project assignment? Respond in the form of a list. (Separate your list into different categories of activities) • In team-based design, documentation and exchange of design information are important. List the characteristics of written and oral communication that will be essential in a team design project.

V.2--TAMU TIDEE Test: 2nd Adaptation Given, as Post-test, to Aero Senior Design Class and a Different Freshman Class in Spring 2002 • Objective: Demonstrate your knowledge of key elements in the engineering design process, teamwork, and communication associated with team-based engineering design. • Assignment: Respond to the following questions/statements as instructed (Time allowed 20-25 mins.) • In a general sense, a process is an ordered set of activities to accomplish a goal. In the space provided, describe your understanding of the engineering design process, in diagram or flowchart form, in complete sentences, or in list form. • What kinds of things should teams do to prepare themselves for effective teamwork in a design project assignment? Respond in the form of a list or in sentences. • In team-based design, documentation and exchange of design information are important. Describe communication qualities and how communication should occur among team members in a design project.

V.3--TAMU TIDEE Test: 3rd Adaptation Given, as Pre-test, to Electrical Engineering Design Class in Fall 2002 • Objective: Demonstrate your knowledge of key elements in the engineering design process, teamwork, and communication associated with team-based engineering design. • Assignment: Respond to the following questions/statements as instructed (Time allowed 20-25 mins.) • In a general sense, a process is an ordered set of activities to accomplish a goal. In the space provided, describe your understanding of the engineering design process, in diagram of flowchart form, in complete sentences, or in list form. • Part A: For a Team-Based Engineering Design Project, what are the processes that contribute to good teamwork? List as many as you can. • Part B: List the qualities that describe effective teamwork. List as many as you can. • C. What are the component characteristics of good communication with regards to the quality of information communicated and with regard to receiving and presenting information?List as many qualities as possible.

V.4--TAMU TIDEE Design Knowledge Test: 4th Adaptation Based on WMU Faculty Feedback – For future use • Objective: Demonstrate your knowledge of key elements in the engineering design process, teamwork, and communication associated with team-based engineering design. • Assignment: Respond to the following questions/statements as instructed (Time allowed 20-25 mins.) • In a general sense, a process is an ordered set of activities to accomplish a goal. In the space provided, describe your understanding of the engineering design process, in diagram of flowchart form, in complete sentences, or in list form. • For a Team-Based Engineering Design Project, what are the processes that contribute to good teamwork? What are the qualities that describe effective teamwork. List and describe as many as you can. • C. What are the characteristics of good communication? (Include information about quality of communication with people external to the design team, as well as internal to tam.) Include as many characteristics as possible.

Our Prototype: The TIDEE Design Knowledge Assessment Scoring Rubric(TIDEE Website, June 2001) Category: Design Process Subcategory: Problem Definition 1 5 No design requirements stated, or few but Goal and requirements defined fully, revised ambiguous at best. over time, address technical and non-technical issues such as performance, cost, reliability, manufacturability, and safety. Subcategory: Evaluation and Decision Making 1 5 Only cursory analysis of ideas; decision made arbitrarily Quantitative and qualitative issues analyzed; appropriate analytical and experimental methods, tools, and information used; decisions based on established criteria Subcategory: Process Development 1 5 Design steps performed linearly without steps towards Process elements repeated, used iteratively to improvement. improve results; design process planned, recorded, and revised for improvement.

Our Prototype: The TIDEE Design Knowledge Assessment Scoring Rubric(TIDEE Website, June 2001) Category: Teamwork Subcategory:Purpose and Goals 1 5 Team goals non existent or poorly defined; members Team driven by clear goals; members know goals clueless about team goals. and how will be assessed; members committed to common purpose. Subcategory: Roles and Responsibilities 1 5 Team members have no clear role assigned or role Members understand responsibility; team defines, ownership; important roles not performed. enforces expectations; members own and fulfill their roles to support team; all important roles covered well. Subcategory: Resource Management 1 5 Team without adequate skills and resources to Team has proper mix, levels of skills needed for Complete project; no effort to obtain needed resources. Project; team uses talents, knowledge, skills of all members; other resources accessed and used effectively.

Our Prototype: The TIDEE Design Knowledge Assessment Scoring Rubric(TIDEE Website, June 2001) Category: Communication Subcategory:Content 1 5 Information incomplete or has obvious errors; Information stated clearly, completely, and overall Reliability questioned. accurately; measures of reliability stated and justified. Subcategory:Relevance to Audience 1 5 Information not relevant to audience as presented. Information in form, appearance, language, and level attractive and understandable to intended audience; understanding communicated well. Subcategory: Value 1 5 Information recorded or communicated has little Information is timely and of great value to team value to design team or others. and to others; support quality design decisions and value-added actions by others.

V.1--TAMU TIDEE Rubric: 1st Adaptation (Fall 2001) -- Used on Fall 2001 Pre-tests , Spring 2002 Post-tests, and Tests completed by TAMU Freshman and Design Profs. 5 Points—Student’s response shows breadth and depth of knowledge, and indicate interrelationships among most (5 or 6 of 6) of the listed elements in the engineering design process necessary to produce design products. 4 Points—Student’s response shows more than moderate knowledge, if it includes 4 of the 6 elements (listed here) of the engineering design process. 3 Points—Student’s response shows moderate knowledge, if it include 3 elements of the engineering design process necessary to produce design products. 2 Points—Student’s response shows little knowledge, if it includes 2 elements of the engineering design process. 1 Point---Student’s response indicates an effort made to describe the engineering design process and one element is indicated but the response lacks coherent organization. 0 Point---Student did not make any effort to define the engineering design process and shows no knowledge of any of the elements listed. Part A: Knowledge of the Engineering Design Process Six specific elements articulated by students. 1. Information gathering (i.e. looking for resources in the library, Web-sites or from the other sources) 2. Problem definition (i.e. understanding problem, goals or requirements) defined 3. Idea generation (i.e. brain storming, creative ideas to improve the design products) 4. Evaluation and decision making (i.e. analyzing ideas, selecting, utilizing, judging and making design decisions based on established requirements) 5. Implementation (i.e. product delivered on time, product satisfy requirements) 6. Process development (i.e. reviewed for improvement)

V.1--TAMU TIDEE Rubric: 1st Adaptation (Fall 2001) -- Used on Fall 2001 Pre-tests , Spring 2002 Post-tests, and Tests completed by TAMU Freshman and Design Profs. Part B: Knowledge of Effective Teamwork Six specific elements articulated by students. 1. Purpose and Goals (i.e. goals, members know goals) 2. Roles and Responsibilities (i.e. members understand roles and fulfill to support teamwork) 3. Team attitude (i.e. polite acceptance and respect for each other, show pride in team and motivated toward team success, commitment) 4. Planning (i.e. time management) 5. Resource management (team uses talents, knowledge, skills of all members, other resources accessed) 6. Operating procedures (i.e. support effective interaction, discipline, listening, decision making etc.) 5 Points---Student’s response (if include 5-6 of 6) shows detailed knowledge of the listed elements of effective teamwork 4 Points---Student’s response shows above moderate knowledge, if include 4 elements (of listed) of the effective teamwork. 3 Points---Student’s response if include 3 elements of the effective teamwork, shows moderate knowledge of the subject. 2 Points---Student’s response including 2 elements of the effective teamwork shows little knowledge of the subject. 1 Points---Student’s response shows little effort in giving the elements of effective teamwork (one element indicated). 0 Points---Student’s response shows no knowledge about the effective teamwork.

V.1--TAMU TIDEE Rubric: 1st Adaptation (Fall 2001) -- Used on Fall 2001 Pre-tests , Spring 2002 Post-tests, and Tests completed by TAMU Freshman and Design Profs. Part C: Knowledge of Effective Communication Five specific elements articulated by students. 1. Structure (i.e. organization, highly understandable, flow of thoughts) 2. Content (i.e. details, key points, clarity of ideas, complete and accurate information) 3. Relevance to audience (i.e. communicated well and understandable to audience) 4. Team attitude (i.e. co-operation, listening) 5. Involvement (i.e. planning meetings, effective interaction between members) 5 Points--- Student’s response (if include 5of 5) shows detailed knowledge of the listed elements of effective communication. 4Points --- Student’s response shows above moderate knowledge, if include 4 elements (of listed) of the effective communication. 3 Points---Student’s response if include 3 elements of the effective communication, shows moderate knowledge of the subject. 2 Points---Student’s response including 2 elements of the effective communication shows little knowledge of the subject. 1 Points---Student’s response show little effort in giving the elements of effective communication (one element indicated). 0 Points---Student’s response shows no knowledge about the effective communication.

V.2 TAMU-TIDEE Rubric: 2nd Adaptation (Spring 2002)Piloted in FC Assessment Workshop with Faculty, at Western Michigan University, Oct 2002 • Knowledge of Engineering Design Process • Six specific elements articulated by students. • Information gathering • Problem definition (i.e. understanding problem goals or requirements defined) • Idea generation (i.e. brain storming, Creative ideas to improve the design products) • Evaluation and decision making (i.e. analyzing ideas) • Implementation (i.e. product delivered on time, product satisfy requirements) • Process development (i.e. reviewed for improvement) • 5 points total: 1 point given for each topic named. • 5 points total: 1 point given for each definition of the main topics • 5 points total:2 points given for listing the topics in the correct order • 2 points given for showing an iteration in the process • 1 point given for showing more that one iteration in the process • Grand Total of 15 Points for Section

V.2 TAMU-TIDEE Rubric: 2nd Adaptation (Spring 2002)Piloted in FC Assessment Workshop with Faculty, at Western Michigan University, Oct 2002 • Knowledge of Effective Teamwork • Three specific elements articulated by students. • Purpose and planning (goals, members know goals, time management, meetings) • Rules, roles and responsibilities (members understand and fulfill roles to support teamwork; team uses talents, knowledge, skills of all members, other resources accessed, and communication) • Team attitude (i.e. polite acceptance and respect for each other, show pride in team and motivated toward team success, commitment, constructive criticism, and communication) • 5 points total for each main category: • 1 point given for each underlying characteristic within the three elements. (Up to 3 points awarded for this part) • 2 points given for examples or more explanation of why it is important • Grand Total of 15 Points for Section

V.2 TAMU-TIDEE Rubric: 2nd Adaptation (Spring 2002)Piloted in FC Assessment Workshop with Faculty, at Western Michigan University, Oct 2002 • Knowledge of Effective Communication • Five specific elements articulated by students. • Structure(i.e. organization, highly understandable, flow of thoughts) • Content (i.e. details, key points, clarity of ideas, complete and accurate information) • Relevance to audience (i.e. communicated well and understandable to audience) • Team attitude (i.e. co-operation, listening) • Involvement (i.e. planning meetings, interaction) • 5 points total: 1 point given for each characteristic that relates to the five elements for communication within the team. • 5 points total: 1 point given for each characteristic that relates to the five elements for communication outside of the team. • 5 points total • 2.5 points for more explanations of communication within the team • 2.5 points for more explanations of communication outside the team • Grand Total of 15 Points for Section

V.3 TAMU-TIDEE Rubric: 3rd Adaptation (Fall 2002)Result of WMU Faculty Feedback – For Use on Fall 2002 Electrical Engineering Design Class Pre & Post tests • Knowledge of Engineering Design Process • Six specific elements articulated by students. • Information gathering • Problem definition (i.e. understanding problem goals or requirements defined) • Idea generation (i.e. brain storming, Creative ideas to improve the design products) • Evaluation and decision making (i.e. analyzing ideas) • Implementation (i.e. product delivered on time, product satisfy requirements) • Process development (i.e. reviewed for improvement) • 5 points total: 1 point given for each of these or similar process actions named. • 5 points total: 1 point given for each addtl “point of information” about a process action. No more than 2 for any one action • 5 points total:2 points given for listing process actions in logical, developmental order similar to that given here. • 2 points given for showing an iteration in the process • 1 point given for showing more that one iteration in the process • Grand Total of 15 Points for Section

V.3 TAMU-TIDEE Rubric: 3rd Adaptation (Fall 2002)Result of WMU Faculty Feedback – For Use on Fall 2002 Electrical Engineering Design Class Pre & Post tests • Knowledge of effective Teamwork • Three specific elements articulated by students. • Purpose and planning (goals, members know goals, time management, meetings) • Rules, roles and responsibilities (members understand and fulfill roles to support teamwork; team uses talents, knowledge, skills of all members, other resources accessed, and communication) • Team attitude (i.e. polite acceptance and respect for each other, show pride in team and motivated toward team success, commitment, constructive criticism, and communication) • 5 points total for each main category: • 1 point given for mention of each underlying contibutor to any of the3 main categories given. (Up to 3 points awarded for this part) • 2 points given for examples or more explanation about how or why a category or any of its contributing elements is important • Grand Total of 15 Points for Section

V.3 TAMU-TIDEE Rubric: 3rd Adaptation (Fall 2002)Result of WMU Faculty Feedback – For Use on Fall 2002 Electrical Engineering Design Class Pre & Post tests • Knowledge of Effective Communication • Five specific elements articulated by students. • Structure(i.e. organization, highly understandable, flow of thoughts) • Content (i.e. details, key points, clarity of ideas, complete and accurate information) • Relevance to audience (i.e. communicated well and understandable to audience) • Team attitude (i.e. co-operation, listening) • Involvement (i.e. planning meetings, interaction) • 5 points total: 1 point given for each characteristic or property that relates to the five elements given with regard to communication within the team. • 5 points total: 1 point given for each characteristic that relates to the five elements given with regard to communication with persons outside of the team. • 5 points total • 2.5 points for more information or explanations about communication within the team • 2.5 points for more information or explanations about communication outside the team • Grand Total of 15 Points for Section

Part II Application of the TAMU-TIDEE Test for baseline and benchmark measurement in Preparation for AERO CRCD “Smart Materials Project” Evaluation.

Application The TAMU-TIDEE Design Knowledge Test was used in assessments which prepared for the evaluation of The CRCD “Smart Materials” Project, and also provided data for use in instrument revisions Baseline and Benchmark Measurements • 2001-2002 AERO Senior Design Class • 2001-2002 Freshman Engineering Class

Comparative Results:2001-2002 AERO Senior Design Students

Content Analysis & Comparison with Rubric: Senior AERO Design Students -- Knowledge of Design Process

Content Analysis & Comparison with Rubric: Senior AERO Design Students -- Knowledge of Teamwork

Freshman Benchmark Comparisons: Beginning vs. End of 2001-2002 Yr. *Different students in Pre and Post test groups

Content Analysis & Comparison with Rubric: Profs. & Freshman (Post-test )Knowledge of Design Process

Content Analysis & Comparison with Rubric: Profs. & Freshman (Post-test )Knowledge of Teamwork

Concluding ObservationsAbout Student Performance Both student groups (freshmen and seniors) • performed well in the pre-test, and improved the most, from pre to post testing, in question 1 (knowledge about the design process). • performed as well or better, in the pre-test, on question 2 (knowledge about teamwork) than on question 1, their performance declined considerably from pre to post testing. • Performed worst in the pre-test and declined the most, from pre to post testing, in question 3- knowledge about communication in team design.

Concluding Observations • Overall, students tended to answer questions as their professors did more than by the Scoring Rubric criteria • The better the students performed from pre to post testing on a question, the more likely they were to respond to questions as their professors did • Many of the professors’ answers were not in line with the Scoring Rubric s criteria • Professors and students both diverged most from the Scoring Rubric s criteria on question 3 (knowledge about communication) and then on question 2 (knowledge about teamwork)

Concluding Observations • Regardless of revisions aimed at improving alignment of student answers with Scoring Rubric s criteria, performance on question 3 ( communication in team design) did not improve • Given the discrepancies, on question 3, between the Scoring Rubric criteria and the answers of professors, we believe that conceptions about what is important and what should be taught about communication in team design, may be different at Texas A&M Univeristy than among the engineering professors who developed the TIDEE design assessment instrument used as a prototype for the TAMU-TIDEE adaptations.

REFERENCES & END NOTES 1 CRCD: Development of an Integrated Multidisciplinary Curriculum for Intelligent Systems. NSF Proposal Number 0088118; P I: D Lagoudas. 2 Davis, D. C. (2001).Transferable Integrated Design Engineering Education (TIDEE), Mid Program Assessment: A Three Part Assessment of Team Based Design for Entering Juniors. http://www/cea/wsu.edu/TIDEE 3 The adaptations and data collections in this presentation involved only Part I of the TIDEE Three Part Assessment of Team Based Design -- Design Knowledge

The AERO CRCD “Smart Materials” Project 1 Preparation for Evaluation