Reviewers comments on our paper submitted to 35th ASEE/IEEE Frontiers In Education Conference

1. Reviewers commentson our paper submitted to 35th ASEE/IEEE Frontiers In Education Conference K. Schützler, K. Bothe

2. Introduction • In 2003 first presentations of current JCSE project state: • Information Technology Interfaces, Cavtat, Croatia • Eurocon, Ljubljana, Slovenia • 1st Balkan Conference in Informatics, Thessaloniki, Greece • 2005: submitted a new paper with current state to 35th IEEE/ASEE Frontiers In Education Conference, Indianapolis Kay Schützler, Baile Herculane, 28.08. - 03.09.05

3. Homepage of FIE 2005 Kay Schützler, Baile Herculane, 28.08. - 03.09.05

4. Topics of interest on FIE 2005 • Accreditation and assessment • Active learning • Capstone and senior design experiences • Computer and Web-based software • Creative design experiences • CSET educational research • Distance learning: Methods, technologies, and assessment • Diversity: Valuing it, achieving it, and teaching it • Entrepreneurship programs • Ethics: Creative ways to teach and assess it • Faculty development • First-year courses and programs • Globalization: Preparing faculty and students • Innovative degree programs and curricula • Innovative uses of technology in the classroom • K-12 initiatives and partnerships • Laboratory experiences: On-site and at a distance • Learning models • Lifelong learning • Nontraditional students • Partnerships (industry, government, university, international) • Service learning • Software engineering • Student retention and persistence • Teaming • Undergraduate research experiences • Women in CSET education • Other (You may submit abstracts and proposals on other topics that address issues at the frontiers in CSET education.) Kay Schützler, Baile Herculane, 28.08. - 03.09.05

5. K. Bothe, K. Schützler, Z. Budimac, K. Zdravkova:Collaborative Development of a Joint Web-Based Software Engineering Course Across Countries Kay Schützler, Baile Herculane, 28.08. - 03.09.05

6. Aims of this presentation • Document the serious and detailed review process of FIE 2005 • Make reviewers comments available for all project participants • Draw some conclusions Kay Schützler, Baile Herculane, 28.08. - 03.09.05

7. January 10, 2005: Abstract submission February 20, 2005: Acceptance/Rejection of abstract March 21, 2005: Full paper submission April 30, 2005: Notification on review results May 30, 2005: Submission of revised paper with modification notes Kay Schützler, Baile Herculane, 28.08. - 03.09.05

8. Excerpt from first accept notification email Dear Klaus, The final recommendation for FIE paper 1106 is: ACCEPT with major changes That means: Each comment of the reviewers must be addressed (even if considered inappropriate) In the comments below, you will see numerical scores for different aspects of your presentation. The scale was: 1 - poor 2 - needs work 3 - good 4 - very good 5 - excellent Kay Schützler, Baile Herculane, 28.08. - 03.09.05

9. Note from Heidi Diefes-Dux(from the organizer‘s site): Two reviewers raise a number of questions/issues that require greater revision than the reviewers indicated - thus my recommendation. As you do have an amount of space left in the paper to addresses these, I recommend that you focus on those that would broaden the usefulness of the paper to a wider audience. In an email to me, please indicate how and where the issues raised by the reviewers are addressed. If you feel that the comments are not appropriate, please indicate that to me as well. Kay Schützler, Baile Herculane, 28.08. - 03.09.05

10. Reviewer 1 Reviewer's recommendation: accept_as_is New ideas: yes Comments on new ideas: It addresses collaborative development of course materials across several countries. Soundness of work: yes Contribution: 3 Comments on work contribution: project management issues Presentation quality: 4, Writing quality: 4, References: yes Gender neutral language: yes, Should anything be deleted: no Completeness of work: no Comments on completeness: still somewhat of a work in progress That’s all we received from this reviewer Kay Schützler, Baile Herculane, 28.08. - 03.09.05

11. Reviewer 2 Reviewer's recommendation: accept_revise New ideas: yes Comments on new ideas: Describes experience with a shared course in software engineering with participants in multiple countries. Approach could be useful for technical education in many locations around the world. Soundness of work: yes Kay Schützler, Baile Herculane, 28.08. - 03.09.05

12. Reviewer 2 (cont.) Contribution: 4 Comments on work contribution: Describes a successful project in which many participants not sharing a language use common course materials to support a course in software engineering. Insight and experience from this project could be of broad general interest, particularly in developing countries and countries partnering with them in education. Presentation quality: 4, Writing quality: 4, References: yes Gender neutral language: yes Should anything be deleted: no, Completeness of work: yes Kay Schützler, Baile Herculane, 28.08. - 03.09.05

13. Reviewer 2 commentsWriting quality This is interesting work and the quality of the writing is fairly good. My questions address relatively minor points, but I encourage the authors to read over the paper again carefully to catch a few remaining awkward sentences, typos, and misused words. The reviewer’s comment We went through the paper as requested and indeed found a few mistakes and even one incomplete sentence. Our answer Kay Schützler, Baile Herculane, 28.08. - 03.09.05

14. skip Reviewer 2 commentsWeb-based course? You describe the course as web-based, but that applies more to the instructors than student participants. (That is, students apparently attend lectures where they see powerpoint slides rather than use machines on a network.) This seems to be inconsistent with the usage of the term in my experience. We agree with reviewer's description of our course, but we still call the course web-based. In section "Overview of project management and course materials" we added "Although students are expected to attend regular lectures, the course as a whole could be called web-based. All documents the students need are published at the course website: the slides in PDF-format, case studies, assignments, tool information, literature recommendations, schedule, etc. Kay Schützler, Baile Herculane, 28.08. - 03.09.05

15. Figure 3: Contributions to the project Translate materials to English Use slides in lectures and give usage reports Add new slides and other materials Design and administrate the project web-site Include new topics Contribute to the assignments Course development Write lecture notes Contribute to the pool of case studies Write review reports Develop slide style guides Translate materials to native language Reviewer 2 commentsContributions from all participants? The abstract mentions a "plethora of interesting contributions" from the diff-erent participants across various universities and countries. Could you give some examples? I'm curious how the course has evolved over time. You allude to such changes but give essentially no concrete examples or details. We added a paragraph in section that gives an example how the contributions changed the course. Besides that, we would like to draw the attention to Figure 3 for additional information on the contributions of all project partners. Kay Schützler, Baile Herculane, 28.08. - 03.09.05

16. skip Reviewer 2 commentsOne comment – many questions: examinations, tools, literature, duration You don't say much about some of the links under course materials, specifically the examinations, tools, literature, and duration links. Are copies of all examinations shared with other participants? What about translation overhead? What kind of tools are currently available, or do you plan to make available in the future? Will the literature link simply give citations of relevant papers, or does this describe internal documents for the course? What duration information is included: just the time required in class to cover each topic or to cover each lecture? We extended section "Course materials" with the new subsections "V. Literature" and "VI. Tools". It covers now all relevant aspects of the materials on the project website. In section "Overview of project management and course materials" we extended the information concerning the Duration link. Kay Schützler, Baile Herculane, 28.08. - 03.09.05

17. Reviewer 2 commentsAssignments Are the assignments linked in (under course materials) labs or homework style problems? What kinds of assignments have been included in the course so far, and what kind of tools or documentation do they require? Again, the missing information is addressed in two new paragraphs in subsection "IV. Assignments" of section "Course materials". "Typical tasks of the assignments include: reviews of requirements specifications, object-oriented modeling practices, performing cost estimations, applying test case determination methods, automated regression testing, and evaluating software metrics." Kay Schützler, Baile Herculane, 28.08. - 03.09.05

18. skip Reviewer 2 commentsConcurrent updates? It seems challenging to let all participants update course mate-rials but to store only a single current file. What if you don't like the changes someone else makes? To address this question we added an extra paragraph at the end of subsection "IV. Update management" in section "Project management to organize collaborative work": "Of course, it is necessary to assess the results of the update process. Modified topics are reviewed by their original author. If there are any issues raised, they will be discussed and solved in a cooperative way between author and modifier." Kay Schützler, Baile Herculane, 28.08. - 03.09.05

19. skip Reviewer 2 commentsCollaboration success measured? Do you have any way of measuring the effectiveness of the class or of this approach to managing a class? To what extent has it reduced the overhead for new participants in creating a new class and getting it up and running? The "Conclusions and future work" section has been extended: "Our original goals to disseminate experience and to reduce efforts in creating a new class on software engineering were reached successfully. Participants at several universities could introduce an approved and mature course with all of its parts (lectures, assignments, case studies, etc.) Both, the many years of experience and the produced materials could be reused. The partner universities in southeast Europe experienced significant reduces (compared to earlier attempts of building new courses from scratch) of the managerial efforts for introducing the course, a benefit that can hardly be overestimated." Kay Schützler, Baile Herculane, 28.08. - 03.09.05

20. Reviewer 3 Reviewer's recommendation: accept_revise New ideas: no Comments on new ideas: Multi-national collaboration and material sharing. Soundness of work: yes Contribution: 2 Comments on work contribution: Giving confidence to others to follow this pattern of working. Kay Schützler, Baile Herculane, 28.08. - 03.09.05

21. Reviewer 3 (cont.) Presentation quality: 5 Writing quality: 4 References: yes Gender neutral language: yes Should anything be deleted: no Completeness of work: no Kay Schützler, Baile Herculane, 28.08. - 03.09.05

22. Reviewer 3 commentsMissing information Extra commentaries by the participants and a retrospective an-alysis of the effectiveness of the transfer process would have been useful. As mentioned above, we discuss the effectiveness of the tran-sfer in our new version of the paper. We do not think that com-mentaries would fit in the structure of the paper. In our opinion, the paper as a whole presents the common experiences of all project partners. Kay Schützler, Baile Herculane, 28.08. - 03.09.05

23. Reviewer 3 commentsInfluence of related work The paper was well prepared and presented, expressed in good English and easy to read, with sufficient reference to earlier activities and relevant publications. But how was the direction and management of this project influenced by the earlier work? To clarify, why our project could not be a simple repetition of the referenced projects, we added a paragraph: "Compared to similar approaches [2, 3, 5, 6] for collaborative preparation and usage of joint course materials, our project contains a couple of unique features: multi-lingual project partners, a strict style for lecture notes, and a strong strategy for project management with review reports, update manage-ment, style guides, etc." Kay Schützler, Baile Herculane, 28.08. - 03.09.05

24. skip Reviewer 3 commentsAddressed the right issues? I was disappointed in the lack of critical appraisal, or pedagogic discussion concerning the effectiveness of the transfer process. I do not think, if I were to embark on a similar multi-national col-laboration, that I would rush to reread the paper for advice and guidance. There is little here which challenges common sense, and no experiential commentary. As mentioned in section "Conclusion and future work", more and more partners joined us over the years and will join us in the future. From our experience the questions raised by new partners are indeed the ones addressed in our paper. Thus, we do not find this comment appropriate. Kay Schützler, Baile Herculane, 28.08. - 03.09.05

25. Reviewer 3 commentsKey information of the paper The topic and information about how to run similar projects could be of genuine value for many teachers in HE across Europe, and may interest US academics too. What key information do the authors hope to relate to readers? What are the principal benefits to be gained from reading this paper? We added a paragraph in section "Conclusions and future work" to explicitly state that our project could serve as an guiding example to similar settings:"Besides that, our project shows that the process of transfer-ring an existing course to other environments (even with dif-ferent languages) can be performed successfully, as long as the issues we presented here are addressed appropriately." Kay Schützler, Baile Herculane, 28.08. - 03.09.05

26. Reviewer 3 commentsStudents cooperation Are students at the partner institutions encouraged to communicate or even cooperate on case studies and assignments? "Up to now, students do not communicate or cooperate across universities." Kay Schützler, Baile Herculane, 28.08. - 03.09.05

27. skip Reviewer 3 commentsEffectiveness of course at partner sites Was any appraisal undertaken to gauge the differences between the effectiveness of the course material when used in the original insti-tution, and that achieved in the acceptor universities? Did student feedback get compared? Were graded assignments and exam ques-tions shared? How was success/failure judged? We extended our paper with another paragraph:"Up to now, students do not communicate or cooperate across univer-sities. However, there is already some kind of indirect cross-university influence from the students. Their solutions to the assignments are compared throughout the whole project and they are evaluating their respective course using a project-wide feedback form. The compari-son of these feedback forms is subject to future work." Kay Schützler, Baile Herculane, 28.08. - 03.09.05

28. Reviewer 3 commentsTranslation to English How was the course material translated from German to English? Who paid for this time consuming work? How were the results checked? How long did it take? Presumably it was an irksome task because of the later reference to automating the work. We changed the first paragraph of section "Project management to organize collaborative work" to cover more details on the managerial background of the project:"Because our project received substantial funding from DAAD, yearly workshops and coordination meetings were possible. Partners from SEE were enabled to stay at the German university several times in order to work on project tasks like translating existing and preparing new materials." Kay Schützler, Baile Herculane, 28.08. - 03.09.05

29. skip skip Reviewer 3 commentsMobility issues and material reuse How much travel, face-to-face meetings for academics, or guest-lecturing was possible during this project? The change of the first paragraph of section "Project management to organize collaborative work" does apply to this question, too. As the courses were ACM compliant, would it have been possible to use an alternative course, already available in the English language? We do not think that using other existing courses would have been appropriate, because the partners explicitly expressed their interest in the particular course available at Humboldt University. We changed the respective sentence in the first section to reflect this situation more. Kay Schützler, Baile Herculane, 28.08. - 03.09.05

30. skip Reviewer 3 commentsMisunderstandings "the aims changed over time" - how, why and in what manner? We think, that this issue has been addressed already in the directly fol-lowing sentences. We made it a bit clearer now, by putting a colon instead of a dot. "…over time, the targets … of the project have changed considerably: With the contributions from participating partners, new materials have been developed as a response to the feedback and suggestions of professors and their students. Due to the additional information the project had to face and overcome organizational challenges to coordinate the work." "special course material problems" - what were these? We assume that this results from a misunderstanding of our bad English. We changed it to "special problems concerning course materials" and again replaced a dot by a colon. Kay Schützler, Baile Herculane, 28.08. - 03.09.05

31. Reviewer 3 commentsPaper from one partner only? The paper appears to have been produced by the senior part-ner, with little evidence of reported experience from the junior, accepting institutions. Views more clearly from all the instituti-ons would have been welcome. We do not consider this comment as being appropriate. We are not able to imagine what kind of evidence the reviewer is requesting. The paper has been produced together by the senior partner and two of the junior partners and contains experience of both sides. Kay Schützler, Baile Herculane, 28.08. - 03.09.05

32. Reviewer 3 commentsCase studies Well chosen case studies are often the most useful material, "borrowed" by colleagues from different universities. Little was said on this subject. Were the two original scenarios specially drafted, or inherited from the original, parent course? Mention is made of the tutors' commercial experience being brought to bear locally for the benefit of the students. How can this resource be distributed successfully to other sites? In subsection "III. Case studies" we added a paragraph:"Presently, we are including two case studies reused from the original course, one from an administrative application and the second from a technical application. The project participants contributed additional well suited case studies. These case studies are not included in the course so far, because teaching materials for them still have to be prepared." Kay Schützler, Baile Herculane, 28.08. - 03.09.05

33. skip Reviewer 3 commentsTechnical problems and recommended literature For the reviewer, the project URL, reference [7], did not work when accessed. An email was dispatched to the principal author. We fixed that problem. Instead of ".../index.html" it has to be ".../index.htm". What text books were recommended for use by students? As mentioned above, we added subsection "V. Literature" to section "Course materials" to give more information on the literature recommendations in our course. Kay Schützler, Baile Herculane, 28.08. - 03.09.05

34. Reviewer 3 commentsConfiguration management Was any project source file control system (CVS?) employed to maintain project consistency? This question is answered in the last paragraph of subsec-tion "IV. Update management" in section "Project manage-ment to organize collaborative work“:"Since most materials are in binary formats and are not changed concurrently there was no urge to apply a complete configuration management. However, previous versions are archived to be used as fallbacks in case of problems." Kay Schützler, Baile Herculane, 28.08. - 03.09.05

35. Final note from Heidi Diefes-Dux(from the organizer‘s site): Klaus Bothe - Thank you for making those changes. I am very satisfied with your responses to the reviewers and I am looking forward to seeing you and your presentation in Indy! Heidi ************************************************************* Heidi A. Diefes-Dux, Assistant Professor Purdue University Department of Engineering Education ************************************************************* Kay Schützler, Baile Herculane, 28.08. - 03.09.05

36. Conclusions • Our project is interesting and useful for third parties • FIE reviewers are interested in quantifiable results • Commonly used real-life case studies are challenging • Still a somewhat open question in our project • Cooperation between students across countries is to be considered seriously • Feedback from partner universities is highly recommended – especially critiques Kay Schützler, Baile Herculane, 28.08. - 03.09.05