Software building

1. Software building Tomáš Kalibera, Tomáš Pop

2. Plan • SW building • Requirements • Typical scenarios • Dependency problems • …. • Tools • Make • Ant • Make – advanced options • Autotools{conf, make, scan,..} • Ivy, Maven, SCons… This lesson Next lesson Tomáš Kalibera, Tomáš Pop

3. What is (for this lecture) software building • Transforming software sources (tree of source files) into executable binary code • In general, it is simmilar to other transformations of files that are carried out by independent applications, such as • Transforming LaTeX document into PDF • Re-sizing and publishing a tree of photographs • Running functional tests • Updating a web site, translating XML source documents into XHTML by XSLT transformer Tomáš Kalibera, Tomáš Pop

4. Typical requirements on software building Try to find some Tomáš Kalibera, Tomáš Pop

5. Typical requirements on software building • Automation • Require no or minimal input from user • Run regularly or based on source code modifications • Portability • Run on all systems supported by the software • Do not require (too many) additional tools, compared to the software built • Allow programmers to implement/configure building easily for multiple platforms Tomáš Kalibera, Tomáš Pop

6. Typical requirements on software building • Efficiency • Do not repeat what has already been done • Employ multiple cores if available • Robustness • Build as much as possible – if there is any error, try to build everything what is not affected by that error Tomáš Kalibera, Tomáš Pop

7. Requirements on tools for software building • Generality • The same tool should be usable for a wide range of applications • Easy to use • Programmers – easy to define how building should be done, extensible • End users – easy to build a program with minimal (or ideally, without any) human intervention Tomáš Kalibera, Tomáš Pop

8. Use scenarios for build tools Try describe typical scenarios (developer, end user) Tomáš Kalibera, Tomáš Pop

9. One-shot build from scratch • Usage pattern • Download sources including the build tool/configuration • Run the tool to build everything from scratch • If something goes wrong, fix the problem, delete everything and start again • Requirements • Portability • For simple programs, a (shell) script would probably suffice • Sometimes: include tests of resulting binary Tomáš Kalibera, Tomáš Pop

10. Requirements on tools for software building • Usage pattern • Build from scratch • Modify some sources and/or configuration • Build again, reusing previously built fragments not affected by modifications • Requirements • Correctness – re-build everything what has to be re-built • Efficiency – do not re-build (much) more than that Tomáš Kalibera, Tomáš Pop

11. Problems to solved in build tools Try to find some Tomáš Kalibera, Tomáš Pop

12. Dependencies • Building consists of running (many) external tools operating on files, in correct order • Correct order must follow inter-dependencies of the source files • i.e. first creating binary object files (.o), then the library (.so, .a) • Or, recompiling a source file (.c) when header it depends on changes (.h) Tomáš Kalibera, Tomáš Pop

13. (Build) Dependencies in C Programs • .h C-preprocessor header files • Pre-compiled header files • .c C source files • .o object files • .so, .a library files • Binary executables preprocessor preprocessor compiler archiver linker Tomáš Kalibera, Tomáš Pop

14. Dependency detection – header deps #include <stdio.h> #ifdef __LINUX__ #include <unistd.h> #endif • Dependencies can be detected by C preprocessor (typically via the compiler, i.e. gcc –M) • Dependencies depend on preprocessor directives • This typically maps to different dependencies for different platforms • Detectors therefore sometimes only detect a superset of possible dependencies Tomáš Kalibera, Tomáš Pop

15. Dependency detection – other C deps • Mission impossible • The information is not explicitly there, developers must provide it • Some reverse engineering is still possible, if needed (legacy systems..) • Debug information in binary object files (.o) may include source files • Object files in a static library (.a) may be listed by their names • In theory, listing symbols in executables might help Tomáš Kalibera, Tomáš Pop

16. (Build) Dependencies in Java Programs • .java source files • .class bytecode files • .jar archiver files compiler archiver This becomes much harder with native code…. Tomáš Kalibera, Tomáš Pop

17. Dependency detection in Java (packages) import java.util.HashSet; import org.sdmt.build.JavaDependencyDetector; • Dependencies can be detected by (some) Java compilers (i.e. Jikes) • Dependencies can be circular • This can only be resolved by Java compiler – files forming a dependency cycle have to be compiled together • Some of these dependencies can also be detected from compiled .class files Tomáš Kalibera, Tomáš Pop

18. which .class files are needed ? • which .class files are needed ? • JAR is in fact a ZIP file • Try jar -xf foo.jar or use WinZip • Detecting dependencies is easy • Listing the files • Contains Classpath entry Tomáš Kalibera, Tomáš Pop

19. C# Assemblies • Assembly (EXE, dll) contains metadata • Reference to other assemblies • Used by JIT compiler • The DLL Hell (Windows XP a Windows 2003 Server) • Several versions of a same assembly can be installed at the same time on one computer • At app start time Windows looks for application manifest – describes required versions of used assemblies (in application.exe.manifest file or in RT_MANIFEST resource embedded inside executable) Tomáš Kalibera, Tomáš Pop

20. Other problems solved in build tools Tomáš Kalibera, Tomáš Pop

21. Getting the sources • Support for downloading (sources, binary dependencies) • HTTP, FTP, SCP, … • e. g. Apache Ivy (next lesson) • Support for version control systems • Checkout/export • Or even commit of the changes Tomáš Kalibera, Tomáš Pop

22. Packaging • Creating a software package • Including metadata • Including documentation • Including binaries for a given system… • Compression of package content • Purpose of the packages • Distribution to end users • Deployment into other applications (plugins, modules…), i.e. EJB, Eclipse plugins Tomáš Kalibera, Tomáš Pop

23. And much more… • Generating documentation • Running or creating tests • Bugs checking by code analysis • Calculating code complexity/good-design metrics Tomáš Kalibera, Tomáš Pop

24. Existing tools Tomáš Kalibera, Tomáš Pop

25. Existing tools - Time-line 1970 (Unix) 2003 (Maven) 1977 (Make) 2010 (Now) 1994 (Autotools) 2000 (Scons, Ant) Image source: http://commandline.org.uk/python/ohloh-and-popularity-programming-languages-free-and-open-source-software/ Tomáš Kalibera, Tomáš Pop

26. The non-Java (mostly C) world • Make and its derivatives • Originally by Stuart Feldman, Bell Labs, 1977 • GNU Make originally by Richard Stallman and Roland McGrath, 1993 • GNU Build System (Autotools) • Autoconf – configuration detector • Automake – makefile generator • Libtool – library creation abstraction tool • Scons • Python • Similar functionality as autotools. Tomáš Kalibera, Tomáš Pop

27. The Java world • Apache Ant • James Duncan Davidson, 2000 (initially intended only for Tomcat) • Today de-facto standard for open source Java projects • Maven • Apache Software Foundation • Support for remote repositories with software components to build • Continuum, Cruise Control • Tools for continuous building Tomáš Kalibera, Tomáš Pop

28. Make Tomáš Kalibera, Tomáš Pop

29. Make overview • Expert system • On input it gets a set of rules and a target to accomplish (build) • Based on the rules only, (not any specific algorithm for the program), it finds a way to construct the target • Target • Something to be done/built, i.e. program binary, object file foo.o, distribution package • Rules • Descriptions that can be converted to dependencies between two files and to commands that transform the files Tomáš Kalibera, Tomáš Pop

30. Not specifying build algorithm is advantage • Support for parallel builds • Make can automatically decide what can be built in parallel (without any changes to earlier well written makefiles) • Support for “building as much as possible” • Make can be run in a mode where it compiles everything that does not depend on a target that cannot be built (not stopping on first error it encounters) Tomáš Kalibera

31. Make from user’s perspective • Make file • Textual file with description of rules, commands are command line utilities (compiler, shell commands, etc) • Build tree • Make decides on what has to be modified by examining modification dates • If file foo.o depends on foo.c, and foo.c is newer than foo.o, foo.o has to be recompiled.. • Make tool • Run “make” to build the default target Tomáš Kalibera

32. Make file example OBJ = ../ant/main.o parse.o all: prog prog: $(OBJ) $(CC) -o $@ $(OBJ) parse.o: parse.c parse.h $(CC) -c parse.c target prerequisity command Example from Peter Miller’s article “Recursive Make Considered Harmful” Tomáš Kalibera, Tomáš Pop

33. List of (GNU) Make file features • Built in variables (CC, FC, …) • Implicit rules • Typical compiler invocations, based on built-in variables (CC, CFLAGS, CPPFLAGS,…) • User functions, conditionals, variables • Internal functions • File-based, text and path manipulation • Remaking make files • Make files themselves can be re-made by make • This is very useful for automated generation of dependencies, that can be described within the make file itself Tomáš Kalibera, Tomáš Pop

34. Make limitations • Portability of make files • Make is standardized by POSIX, but not all systems are POSIX compliant… • GNU Make runs everywhere • Writing make files to run both on non-UNIX systems (such as Windows) and UNIX is however a challenge • Inability to describe more complex dependencies than between two files • This matters i.e. for CORBA idl compiler that generates 2 files from a single one • Some argue that Make syntax is not user friendly • … and propose XML syntax that is much worse Tomáš Kalibera

35. Make limitations • Especially with older make implementations, specifying all the dependencies by hand was too much work • Implicit rules help • Automake (and other makefile generators) overcome this issue • More later… Tomáš Kalibera, Tomáš Pop

36. Apache Ant Tomáš Kalibera, Tomáš Pop

37. Ant summary • Java based • Primarily for compilation of Java programs • Wide support for programs and tools common in the Java world (EJBs, JSPs, JUnit, …) • Easily extensible in Java • Main features • Allows very portable scripting of building for Java applications • Weaknesses • Syntax hard to read (XML) • Limited extensibility in dependencies Tomáš Kalibera, Tomáš Pop

38. Ant definitions • Project • Represents the application to be built • Target • A name for a goal required for building (compilation, packaging, generating documentation) • Can depend on other targets • Task • An executable code fragment (i.e. execution of Java compiler) • Property • Name/value pair (i.e. for the base directory) Tomáš Kalibera

39. Ant build file structure <project name="MyProject" default="dist" basedir="."> <description> simple example build file </description> <property name="src" location="src"/> <property name="build" location="build"/> <property name="dist" location="dist"/> <target name="init"> <mkdir dir="${build}"/> </target> <target name="compile" depends="init“ description="compile the source " > <javac srcdir="${src}" destdir="${build}"/> </target> … </project> This is a shortened version of an example from Apache Ant documentation Tomáš Kalibera, Tomáš Pop

40. Ant tasks • Compilation • Javac, jspc, rmic, depend, … • Archiving • Bzip2, zip, gzip, jar, rpm, tar, cab, … • Remote communication • ftp, scp, sshexec, sending emails, … • Testing • JUnit • Scripting support • JavaScripts, Java 1.6 based, … • File-based operations • Platform independent manipulation with files and directories Tomáš Kalibera, Tomáš Pop

41. Source dependency resolution • Recompile everything from scratch • Just run the compiler on all sources all the time • Re-use class files compiled previously • “depend” task, deletes .class files obsoleted by modified sources • Dependencies are read from class files • Some dependencies are however not discovered Tomáš Kalibera

42. Links • Ant • http://ant.apache.org/manual/index.html • Make • http://www.gnu.org/software/make/manual/make.pdf • http://aegis.sourceforge.net/auug97.pdf Tomáš Kalibera, Tomáš Pop

43. Problems • So far, so good… Tomáš Kalibera, Tomáš Pop

44. Problems: 1) Build dependencies in Fortran • .h C-preprocessor header files • .F90 module sources • .mod compiled module info • .F90, .F fortran sources • .o object files • .so, .a library files • Binary executables preprocessor compiler compiler archiver linker Tomáš Kalibera, Tomáš Pop

45. Problems: 1) Build dependencies in Fortran • C sources have headers • Something not generated, but needed for compilation of sources • header dependencies influence what has to be recompiled • Java sources have packages • Something generated by compiler from sources, needed for compiling other sources • package dependencies influence the order of compilation • Fortran sources have both • Headers and modules Hardly could be handled by tools like Ant Tomáš Kalibera, Tomáš Pop

46. Problems: 2) Changes in source structure • Make • It is necessary to modify Makefile when changing structure • Building too much • Not all dependencies covered Tomáš Kalibera

47. Problems: 3) Recursive makefiles • Idea and figures from paper: Petter Miller Recursive Make Considered Harmful • Unix Word often using recursive makefiles • Consider following structure: MODULES = App Parse all: For dir in $(MODULES); do \ (cd $$dir; ${MAKE} all; \ done • MyApp • Makefile • App • Makefile • main.c • Parse • Makefile • parse.c • parse.h • parse.y all: prog prog: parse.o ../App/main.o $(CC) -o $@ $< parse.o: parse.cparse.h $(CC) -c parse.c parse.cparse.h: parse.y $(YACC) -d parse.y all: main.o main.o: main.c\ ../Parse/parse.h $(CC) -I../Parse -c main.c Tomáš Kalibera, Tomáš Pop

48. Problems: 3) Recursive Makefiles MODULES = App Parse all: For dir in $(MODULES); do \ (cd $$dir; ${MAKE} all; \ done • MyApp • Makefile • App • Makefile • main.c • Parse • Makefile • parse.c • parse.h • parse.y all: prog prog: parse.o ../App/main.o $(CC) -o $@ $< parse.o: parse.cparse.h $(CC) -c parse.c parse.cparse.h: parse.y $(YACC) -d parse.y all: main.o main.o: main.c\ ../Parse/parse.h $(CC) -I../Parse -c main.c • Q: What happen if parse.yis changed and incremental build (top level Makefile) is run? Tomáš Kalibera, Tomáš Pop