Process and Open Source Software

1. Process and Open Source Software Walt Scacchi Institute for Software Research UC Irvine http://www.ics.uci.edu/~wscacchi/Presentations/Process/Informatix-Process-Lecture.ppt

2. Overview Core concerns Software process and process modeling Backstory/case study History of software process Current challenges in software process Open source software processes Best practices

3. Core concerns Process(es)? What to do and how to do it Tasks, inputs, outputs, performers, tools Performers enact tasks using tools to transform inputs into outputs Tasks are partially ordered Tasks, inputs, performers, tools may have pre-conditions Tasks, outputs, performers may have post-conditions Organize and coordinate who does what, (with what resources), when, where, how, and why Formal modeling of processes enables systematic querying and reasoning about complex processes

4. Core concerns Process modeling Understanding, analyzing, and (re)designing software development processes Models may be either Prescriptive—what should be done and how to do it Proscriptive—what might be done and how Descriptive—what was done and how (historic record)? Models may be either Narrative Semi-structured (form-based; hypermedia-based) Graphic (attributed flow graphs)? Formal (language-based process programs or specifications)? Families of models inter-related through meta-model Meta-model provides model ontology and process epistemology

5. Backstory/case study Major TelCo wants to develop broadband multi-media telecommunications system Anticipates $1B software development, up to 1500 software developers working 2-3 years Seeks industrial/IT partners to provide supporting infrastructure to reduce risk IT partner wants to showcase new “process support technology” products as sales lead IT partner brings in academic research team to analyze and advise TelCo on “process issues”

6. Backstory Academic team, IT partner, and major TelCo jointly elicit, capture, codify (model) and inter-relate TO-BE system development process. Academic team employs IT partner’s products to present results of their “process analysis” IT partner considers its process-centered product major success to help large sales engagements Academic team view of their effort -- a major success for publication, but...

7. A complex organizational process (and model):  a decomposition-precedence relationship view (19 levels of decomposition, 400+ tasks)?

8. Backstory Academic team suggests overall process will not succeed -- too complex, too much delegation, problematic hand-offs (“throwing it over the wall”)? TelCo and IT partner dismiss academic team Less than one year later, IT vendor abandons process technology product Two years later, business press reports TelCo experiences major project failure and losses greater than $200M, and no system.

9. Software process model history Process flow charts Software life cycle (“waterfall” diagram) Incremental development Spiral model Process programs (formal specification) Capability maturity models and industrial/military standards Not process models--only focus on what to do, but not how to do it.

14. Recent software process challenges Distributed, decentralized, and/or global software development Process improvement Process design optimization or redesign Continuous process improvement (learning) Articulating software evolution processes Open source software development processes Process discovery (semi-)automated extraction of process events, conditions, timestamps, and other meta-data from software development artifacts

16. Free/Open Source Software Development Processes and Practices

17. What is free/open source software development? Free (as in “freedom”) vs. open source Freedom to access, browse/view, study, modify and redistribute the source code Free is always open, but open is not always free F/OSSD is not “software engineering” Different: F/OSSD can be faster, better, and cheaper than SE in some circumstances F/OSSD involves more software development tools, Web resources, and personal computing resources

18. OSS Development Models Free Software (GPL) Open Source (BSD/MIT, Mozilla, Apache) Corporate Source (Hewlett-Packard) Consortium/Alliance (OSDL, SugarCRM) Corporate-Sponsored (IBM-Eclipse, Sun-Netbeans, Sun-OpenOffice, HP-Gelato) Community Source (Sakai, Westwood)

19. OSSD Project Characteristics OSS Developers are always users of what they build, while OSS users (>1%) are also OSS developers Requires “critical mass” of contributors and OSS components connected through socio-technical interaction networks OSSD projects emerge/evolve via bricolage Unanticipated architectural (de)compositions Multi-project component integrations OSSD teams use 10-50 OSSD tools to support their development work

20. OSSD Project Characteristics Operational code early and often--actively improved and continuously adapted Post-facto software system requirements and design OSSD is not Software Engineering OSSD has its own “-ilities” which differ from those for SE Caution: the vast majority of OSSD projects fail to grow or to produce a beta release.

21. SourceForge.net

22. Google Summer of Code 2007

26. Evolutionary redevelopment, reinvention, and redistribution in OSS One evolutionary dynamic of OSSD is reinvention Reinvention enables continuous improvement OSS evolve through minor mutations Expressed, recombined, redistributed via incremental releases OSS systems co-evolve with their development community Success of one depends on the success of the other Closed legacy systems may be revitalized via opening and redistribution of their source When enthusiastic user-developers want their cultural experience with such systems to be maintained.

28. Configuration management and work coordination in OSSD Use CM to coordinate and control who gets to update what part of the system/online artifacts. Many OSSD projects use CVS (single centralized code repository with update locks) and frequent releases (daily releases on active projects) Linux Kernel: Git (multiple parallel builds and release repositories) Collab.Net and Tigris.org: Subversion (CVS++) Apache: Single major release, with frequent “patch” releases (e.g., “A patchy server”) GNU arch seeks to develop Free CM unification

29. Project management and career development in OSSD OSSD projects self-organize as a meritocractic role-hierarchy and virtual project management Meritocracies embrace incremental innovations over radical innovations VPM requires people to act in leadership roles based on skill, availability, and belief in project community OSS developers value freedom of choice and expression. want to learn about new stuff (tools, techniques, skills, etc.), have fun building software, exercise their technical skill (e.g., developing reusable code), try out new kinds of systems to develop, and/or interconnect multiple F/OSSD projects

30. A meritocractic role hierarchy for OSSD

32. Socio-technical and cultural evolution processes in OSSD New kinds of OSSD processes under study Joining and contributing to a project in progress Role-task migration: from project periphery to center Alliance formation and community development Independent and autonomous project communities can interlink via social networks that manipulate objects of interaction Enables possible exponential growth of interacting and interdependent community as socio-technical interaction network

33. Discovering the what and how of OSSD processes Practitioner reflections and anecdotes Surveys Ethnographic field work (virtual ethnography) Mining OSSD project repositories Multi-modal modeling, analysis, and validation of OSSD processes

36. NetBeans

37. Formal model of an OSSD process coded in PML (excerpt) ... sequence Test { action Execute automatic test scripts { requires { Test scripts, release binaries } provides { Test results } tool { Automated test suite (xtest, others) } agent { Sun ONE Studio QA team } script { /* Executed off-site */ } } action Execute manual test scripts { requires { Release binaries } provides { Test results } tool { NetBeans IDE } agent { users, developers, Sun ONE Studio QA team, Sun ONE Studio developers } script { /* Executed off-site */ } } iteration Update Issuezilla { action Report issues to Issuezilla { requires { Test results } provides { Issuezilla entry } tool { Web browser } agent { users, developers, Sun ONE Studio QA team, Sun ONE Studio developers } script { <br><a href="http://www.netbeans.org/issues/">Navigate to Issuezilla </a> <br><a href="http://www.netbeans.org/issues/query.cgi">Query Issuezilla </a> <br><a href="http://www.netbeans.org/issues/enter_bug.cgi">Enter issue </a> } } …

38. PML validation analysis

41. Best practices Processes with explicit process models are easier to manage, analyze, improve, distribute, and reuse New/ unfamiliar software tools and techniques are best candidates for software process support Process meta-modeling enables process life cycle engineering and formal reasoning about processes OSSD is a community building process not just a technical development process F/OSS peer review creates a community of peers OSSD processes often iterate daily versus infrequent singular (milestone) Software Life Cycle Engineering events OSSD: frequent, rapid cycle time (easier to improve) vs. SLC: infrequent, slow cycle time (harder to improve)

42. Best practices Easiest to improve a process that is formally modeled Process management and improvement have been one of the most enduring practices in Software Engineering for improving productivity and quality, and to reducing cost and risks.

43. Research opportunities

44. Research opportunities FOSSD is poised to alter the calculus of empirical SE Software process discovery, modeling, and simulation Repository mining can support software visualization, refactoring/redesign studies Comparison of SE versus FOSSD approaches to software inspection and peer review

45. Research opportunities Based on results from individual motivation, participation, role migration, and turnover in FOSSD projects, SE world would benefit from empirical studies that examine similar patterns in conventional software development projects Is FOSSD more fun, interesting, and rewarding than SE?

46. Research opportunities Conventional software cost estimation techniques (e.g., “total cost of operation”) slight/ignore social capital and socio-technical resources Miscalculation of total resources and capabilities that affect predicted/actual costs of software development or FOSSD

47. Research opportunities Results from study of cooperation, coordination and control in FOSSD Virtual project management and role migration processes can provide a lightweight approach to SE project management Unclear whether proprietary software projects willing to embrace VPM

48. Research opportunities Alliance formation and social networking results suggest SE projects operate at a disadvantage compared to FOSSD projects SE projects tend to produce systems whose growth/evolution is limited FOSSD projects can produce systems capable of sustained exponential growth/evolution of both software and developer-user community

49. Research opportunities How best to encourage the emergence of a social movement that combines best practices of FOSSD and SE Consider participation or study of open source software engineering (OSSE) projects at Tigris.org OSSE seeks to combine SE and FOSSD tools, techniques, and concepts

51. Detailed study report available W. Scacchi, Free/Open Source Software Development: Recent Research Results and Methods, in M. Zelkowitz (ed.), Advances in Computers, Vol. 69, 243-295, 2007. http://www.ics.uci.edu/~wscacchi/Papers/New/Draft_Chapter_Scacchi.pdf

52. Acknowledgements Project collaborators: Mark Ackerman, UMichigan, Ann Arbor Les Gasser, UIllinois, Urbana-Champaign John Noll, Santa Clara University Margaret Ellliot, Chris Jensen, UCI-ISR and others ar ISR Funding support: National Science Foundation: #0083075, #0205679, #0205724, #0350754 and #0534771. Digital Industry Promotion (DIP) Agency, Global R&D Collaboration Center, Daegu, South Korea No endorsement implied.