Goal of the White Paper

1. White Paper On Ownership\Sharing/Promoting Of Data, Testbeds, And ArtifactsVic Basili, Marv Zelkowitz, Dag Sjøberg, Philip Johnson, Tony Cowling

2. Goal of the White Paper Provide guidance and discussion on the following questions: • How should data, testbeds, and artifacts be shared? • What limits should be placed on who can use it and how? • How does one limit potential misuse? • What is the appropriate way to give credit to the organization that spent the time and money collecting the data, developing the testbed, building the artifact? • What are the privacy issues? • Once shared, who owns and maintains the combined asset? We would like to receive comments and feedback from the ISERN community.

3. Various Case Studies • Each study has a different view of what “intellectual property” they own, share or want to propmote: • Project Data from a variety of projects • Sets of experiments, various artifacts used for experimentation, and resulting data • Repository of qualitative and quantitative evidence about the effect of processes in many contexts • Models and data from dependability requirements elicitation and analysis, non-trivial technology evaluation testbeds. • Application of a Data Collection Tool & the set of data collected using it • Potentially confidential information about developers and clients based upon the results of experiments

4. Various Case Studies • For each study we try to answer the following questions: • What is the intellectual property? • Who are the participants? • What is the study about? • What was the goal of the project? • What kinds of data were collected and/or artifacts developed? • How many users were there, if known? • What are the primary problems associated with the data ownership and how were they dealt with? • Please share your own idea and experiences with us

5. Study 1. NASA SEL Intellectual Property: Project Data from a variety of projects Participants: UMD, CSC, NASA/GSFC Description: SEL studied software development in the Flight Dynamics Division of NASA/GSFC from 1976 to 2002 and collected data on several hundred projects Goals: Improve the quality of ground support software.

6. Study 1. NASA SEL Data collected: - Resources expended, changes and defects, product characteristics, processes applied, conformance to those processes on several hundred projects. - Various project characteristics also collected, e.g., context variables. - The artifacts developed were operational NASA systems. Models of cost, schedule, and quality were built using this data. - Local models built to characterize the environment, evaluate the effects of technologies, predict cost and schedule Users: Data given away freely, artifacts shared. Data resulted in over 250 publications from UMD alone.

7. Study 1. NASA SEL Data ownership issues: • data often misused and misinterpreted, insufficient number of context variables, • results of analysis often not known or shown to the original data owners, • time spent organizing/making data available to requesters/answering questions • opportunities for feedback and meta-analysis were often lost. Misuse of data a problem for requestors/community: questionable results published. Later mode: to use data, spend time at SEL, interacting/understanding nature of the data. SEL closed in 2002, most of collected data potentially lost - no long lived repository replacing the SEL

8. Study 2. Software Analysis Experiments Intellectual Property: Sets of experiments, various artifacts used for experimentation, and resulting data Participants: UMD, IESE, Brazil( URD, USP, US), Norway, … Description: Multiple organizations ran an experiment to evaluate the effectiveness of reading as a defect detection technology. (mid 1980s) Goals: Originally: evaluate effects of reading vs. testing on code. Later: evolve reading techniques based on empirical evidence applying the techniques in a variety of environments. Major foci included the development of techniques for reading requirements documents/ object oriented designs.

9. Study 2. Software Analysis Experiments Data collected: - Several artifacts (fault seeded code/requirements documents) developed, - Data collected on defects detected by various techniques, or versions of techniques, applied to the artifacts. User: Several ISERN members on several continents - Reusing artifacts allows replication of the original study by multiple sources and promotes empirical work , making it easier for researchers to get started. Several groups replicated the original reading vs. testing experiment, the perspective-based reading experiments.

10. Study 2. Software Analysis Experiments Data ownership issues: - Difficult to replicate the original work without a certain amount of implicit knowledge from the original researchers  time and money in sharing results, but if not shared badly replicated projects or non-combinable results. Possible original researchers will not know about follow on work and therefore cannot help or critique the results. Counter example - the Reader’s Project, collaboration of original experimenters and researchers in Brazil. - Experiments replicated, meta-analysis performed, and recognition of problems with laboratory manuals identified, e.g., tacit information not in laboratory manual. -Joint funding received from both countries. - Results: successful replications, learning about replication, improvement of the artifacts, laboratory manuals, etc.

11. Study 3. CeBASE • Intellectual Property: Repository of qualitative and quantitative evidence about the effect of processes in many contexts • Participants: UMD, FC-MD, USC, … • Description: NSF sponsored project (2000), Center for Empirically-Based Software Engineering (CeBASE) to develop and provide a repository of “experiences” associated with the application of various software development methods, techniques to create hypotheses, qualitative and quantitative models, other form in aggregated experiences. • Goals: Create shared repository of empirical information on the effects of applying a variety of techniques, methods and life-cycle models.

12. Study 3. CeBASE • Data collected: Experiments performed at UMD/USC. Evidence from a variety of sources. Hypotheses published for evolution. • Original focus: defect detection methods, COTS developments, and Agile methods. Variety of data from the experiments. • Users: Original idea was to get the ~ 30 collaborators to provide “experience” in the form of qualitative/quantitative evidence on the effects of the technologies, but not much interaction. • Actual: Readers of the papers and eWorkshop participants questioning and evolving the hypotheses • Result: no monitoring, anyone can use the experience base without the owners knowing about it and providing feedback

13. Study 3. CeBASE • Data ownership issues: • Who pays for the cost of maintaining the experience base? • Who has access to analyze and synthesize and create new knowledge? • How do we assure they wpuld provide quality data, analysis, and new knowledge? • The original experience base maintenance was supported by the NSF grant, but once the grant ended, maintenance of the EB became the responsibility of one of the CeBASE participants with the need to keep the EB alive. Application for infrastructure funding from NSF was reviewed well important but not funded • EWorkshops are still used but the topic depends on funding available

14. Study 4. HDPC • Data collected: To reduce the risk of applying these technologies to live systems before they are demonstrated effective, we built non-trivial experimental testbeds (SCROVER, TSAFE) to compare the effectiveness of various techniques for improving software dependability in isolation and in combination. • Testbed artifacts include requirements (functional and non-functional), design artifacts, test cases, defect seeded versions of all the artifacts, results of applications for various techniques, etc. The artifacts will evolve over time to be effective for new technologies • Users: Testbeds currently used by a small set of researchers to test out their techniques.

15. Study 4. HDPC • Intellectual Property: Models and data from dependability requirements elicitation and analysis, non-trivial technology evaluation testbeds. • Participants: UMD, FC-MD, USC as subcontract to CMU, plus several other universities (MIT, UW, …) • Description: The High Dependability Computer Program sponsored by NASA/Ames was interested in reducing the risk of applying various technologies for building dependable systems by evaluating their effectiveness and maturity • Goals: Define models of dependability and assess the ability of new research techniques to support the development of highly dependable systems for NASA. Example applications: Mars Scientific Laboratory, Earth Observatory System Data Information System.

16. Study 4. HDPC • Data ownership issues: At the moment, there is more control as the experimental environment users need to interact at some level with the testbed developers. • But should the testbed be released on the web and what happens then? • Also, the testbed needs to evolve based upon the needs of the different technologies to have different artifacts for analysis. Who will pay for this? • How will the testbed and the results be maintained? • How should the artifact creators be acknowledged?

17. Study 5. Hackystat • Intellectual Property: Application of a Data Collection Tool and the set of data collected using it • Participants: University of Hawaii • Description The Hackystat project explores automated collection and analysis of software engineering metrics. Sensors attached to individual development tools unobtrusively collect data about product and process and perform analysis • Goals: • Current – provide developers visibilty into their software development • Long term: Convert metric repository to a representation that provides information of general utility to the software development community.

18. Study 5. Hackystat • Data collected Examples of sensor data include: activities (compilation, file editing, etc.) within an editor (such as Emacs, Eclipse, JBuilder, etc.), invocation of unit tests and their results (using a test framework such as JUnit), size/complexity of the system (in LOC, methods, classes, operators, etc. using a tool such as LOCC), configuration management events (such as commits and lines added/deleted using a tool such as CVS), test case coverage (using a tool such as JBlanket), software review process and products (time spent doing review, issues generated, etc. using a tool such as Jupiter), and defect management (using a tool such as Jira). • From this raw sensor data, higher level abstractions can be built regarding the trajectory of development, and analyses can be performed to look for trends in sensor data and co-variance over time. • Users: Over 200 users have sent data to the public Hackystat server since the summer of 2001 when it was first made available. Approximately 1.5 GB of raw sensor data, representing over 10,000 person-days of development currently exist on this server.

19. Study 5. Hackystat • Users In addition to the 200 active users of the public Hackystat server, several groups have installed their own hackystat server and are collecting their own data, including Sun Microsystems, the University of Maryland, Vanderbilt University, and the University of New Mexico. • Data ownership: There are two immediate challenges: • (1) Privacy: What kind of ‘data scrubbing’ process should be performed on the data to allow its publication without revealing identifying details? • (2) Context: Without the addition of some demographic information about the developers, the product under development, and the context (process), it is unclear how the larger community could obtain value from the Hackystat data. • In summary, it appears that we must simultaneously add and subtract information from the current Hackystat database to make it usefully publishable.

20. Study 6. Sheffield Software Engineering Observatory • Intellectual Property: Potentially confidential information about developers and clients based upon the results of experiments • Participants: University of Sheffield • Description Student teams develop software projects for real clients under conditions as industrially realistic as is possible. One set of projects is with second-year undergraduates, another set is with masters students are in masters. Teams compete to build a system that will best meet the client’s needs. • Goals: Allow experimental comparison of complete methodologies, focus so far has been comparing traditional methodologies with agile ones

21. Study 6. Sheffield Software Engineering Observatory • Data collected: (1) Repository of project artifacts: management plans, requirements documents, design documents, test plans, test harnesses, code, etc., and evaluation documents. (2) Basic quantitative development data, e.g., effort in each activity.(3) qualitative data about interactions between team members, attitudes towards work, supplemented by quantitative data from tests such as the MBTI, Warr’s well-being measure, PANAS and the workgroup cohesion measure. • Users: Currently the raw data has been provided to one other group of researchers. We envisage that other researchers may wish to share the data in future, and we also see a need to produce summaries of the results in a form that could be useful to industry.

22. Study 6. Sheffield Software Engineering Observatory • Data ownership issues: The two key issues surrounding the release of raw data are those of client and developer privacy. • For each client the raw data contains much information about the nature of their business that could be commercially sensitive. Any release of raw data would have to be on the basis of the researchers to whom it was released signing a NDA, and this would have to be applied transitively. • Similarly, for each student developer the raw data contains information that they might well wish to keep private

23. Discussion items on sharing data • Based upon the above experiences, we discuss the following attributes related to sharing: • permission to use the items, e.g., write a white paper what they plan to do, ask permission, … • credit the original developer with the work involved, (what should they reference?) and • feedback on the results of use as well as problems with using the artifact or data. • protection of the data and artifacts like privacy, safety, etc. that need to be considered. For example, the data was most likely collected from people who were assured anonymity – how should this be handled? The owner of the artifacts may be interested in the opportunity for • collaboration: Where and how is real collaboration is expected? • maintenance: How do we keep the information alive and available?

24. Permission Does one have to request permission to use the material? Is it simply publicly available? What should be the rules? • If publicly available, should one provide some form of controlled access to the artifacts? How? • Request use of the artifact with a commitment to provide feedback (method, results, other data) and reference the items. • Restrict access by requiring that the requestor write a one-page proposal to the data owner, e.g., on the basis of such a one-page proposal, a written contract should be established. • Then the item can be used: • Freely, in the public domain • With a license • With a service fee for use (by industry) to help maintain the data

25. Credit How should the original group gathering the data or developing the artifact be given credit? What would be the rewards for the artifact/data owner? What credit should be given? • Type of credit should be related to the amount of interaction. • co-authorship if some level of collaboration • acknowledge and reference the data owner if it is used without the support • Reference the paper that used them or some independent item where the artifact itself exists a reference. It is also possible that some form of “associated” co-authorship might be appropriate • Several issues arise here: • Who is the owner/main reference? Is it the first author of the "main" paper? Is it all the authors on the "main" paper? Is it a project or group manager? Is it the project or the group? For example, it might suffice that one person from the owner group is given the credit by the requestor. • What happens if the owner quits or moves to another organization? What happens if the group disbands? Would the owner then be the same person or another person in the original group? (Probably the latter.)

26. Credit • A side issue is internal interactions. • Experiments require a team of people. (Can a large experiment result in a single author paper?) • How is authorship of such work decided? • Does everyone who contributes get on the author list? • What should be the order of authors? • Can we provide some form of guideline here? For example the University of Maryland has been criticized for having too many authors on a paper. This is common practice in experimental physics (everyone from the builder of the instrument to the original experimenter can be an author of a paper) but not accepted practice in computer science.

27. Collaboration • The requestor keep the option open for collaboration on the work. Some options are suggested above. • The Reader’s project has been an excellent example of collaboration but required funding on both sides. Funding agencies are often looking for “new” ideas and so it is often difficult to be funded for a continuing operation. (Not so easy between US and Europe) • What options are there for funding collaborations? (If collaboration is not desired by the owner of the artifacts, what are the rights of the requestor? It is probably too strong to require collaboration as a requirement for any requestor.)

28. Feedback • At the very least, by requiring some form of permission, there is a sense that the originator of the materials knows of someone is using their materials. • However, some form of feedback can act as payment, • updated versions of artifacts, • data so it can be used in some form of meta-analysis, • some indication of the effectiveness of technology on the experimental environment. • (Maybe this can be part of the original agreement. ) • A related issue is assuring that the quality of the data, analysis, and new knowledge being returned to the originator is acceptable and consistent within the context of the original experiment.

29. Protection • How does one limit potential misuse? How does one support potential aggregation and assure it is a valid aggregation? • What is required on the originator’s side? • Should they be allowed to review results before a paper is submitted for external publication? Does the artifact owner have any rights to stop publication of a paper with invalid results based upon the original artifacts or is the “marketplace of ideas” open to badly written papers? • Who has the rights to analyze and synthesize and create new knowledge based upon the combined results of multiple studies? How does one limit potential misuse? • What are the goals or requirements of both artifact owner and user for collaboration.

30. Protection • How does one deal with proprietary data? Even when data is not proprietary, how does one assure anonymity of the data. What about privacy concerns? • How do we “clean” the data? Whose responsibility is it? • Do users need to sign some form of agreement? • How do we get corporate data to be publicly available?

31. Maintenance • One critical practical issue is the maintenance of the experience base of data or artifacts. • Who pays for the cost of maintaining the experience base? • CeBASE applied to NSF to support the infrastructure, although rated highly competitive, it was rejected • Possibilities here toward maintenance: • (1) Owner of the data (won’t work since few have such resources) • (2) Users via a licensing fee (may work, but costs will limit use; researchers won’t generally pay for something they view as a “free resource” • (3) Everyone via an open source arrangement.

32. Potential Solutions • Open Source • Anyone is able to obtain anything from the library and propose changes to any artifact in the library. • There are rules that anything modified from the library is part of the open source product and is freely available to anyone. • Companies make money by providing these basic products and then sell proprietary enhancements to the basic product (e.g., Red Hat Linux). • Can the open source mechanism be used to maintain and enhance software engineering measurement and experimentation artifacts? • Is there a commercial market that will allow the free-based open source mechanism to work?

33. An economic utility model • Define one unit of empirical data (UED). • A data collection or an artifact is worth so many UEDs. A single class experiment may be 1,000 UEDs. A collection like the NASA/SEL database may be large - say 100,000 UEDs. • A license to use a data base might be a fixed number of UEDs - say 100 UEDs or perhaps 10% of its value in UEDs to use it on one experiment or perhaps 500 for an unlimited license.

34. An economic utility model • The licensee can pay for the license in several ways: • Co-authorship of a single paper may be worth 50 UEDs or maybe 100 UEDs. According to IEEE guidelines, coauthors to a paper must contribute to the intellectual content of the paper. So co-authorship is not simply sold, but if we are doing this honestly, the coauthor has to contribute to the paper and it fosters collaboration. • Acknowledgement of the artifact owner in paper should probably be a requirement and not a reimbursable cost. • Feedback of results to owner of data may be worth 50 UEDs as part of the payment. • We might also put a monetary cost on UEDs such as $1= 1 UED so the artifacts can be licensed for $100 from owner with no other requirement on sharing or feedback. (Or perhaps $50 for use of data with requirement to feed back results, etc.)

35. An economic utility model • Note: We may need different models for different phases. To begin with, it may be feasible for the originator to be involved in new papers. However, the (active) co-authorship model doesn’t scale. If the originators are successful in that there will be a lot of requests, then active co-authorship may not be possible. On the other hand, that will only happen after several cases with co-authorships. During that experience, the places that were in need for extra support may have been identified and already rectified or documented well enough for new users. • If the data were valuable, then there would be a market in using it with some collaboration resulting and some funds being transferred. The transferred funds would be used to maintain the data. Very little of existing data we have today would survive since it is so full of holes. Few would actually pay for it. But a growing market in data would encourage developers to be more careful in collecting it. (Note, we need some guidelines for how to describe data, including context variables and methods for data collection, which will depend on the kind of empirical study.)

36. An economic utility model • This doesn't handle maintenance, permanence, permission, or protection. Databases come and go, faculty change jobs and retire. Can we really provide a perpetual storage for this data, although it would be nice. However, after 10 or 15 years it probably is not really very relevant any longer. • The dream of getting an agency such as NSF to maintain a data repository is a dream and unlikely to occur for many years - if ever. • However, why shouldn’t governments be interested in contributing? For example, CERN, the European Organization for Nuclear Research in Switzerland has been funded by its member states (now 20) for more than 50 years. Software engineering is probably more important to society today than is nuclear physics. But the politics needed to attract long-term international funding for SE is of course formidable … • Note: Even if actual cash transfer is not achieved, the model is very interesting in that it makes the value of artifacts and data prominent.

37. Conclusion • The ISERN community has the need to raise the problems associated with experimentation and the sharing of materials and has the visibility in the community to be listened to. We can use EMSE as a potential sounding board for opinions.