1 / 51

Continuous Integration in the Cloud with Hudson

Learn how to offload work from people to computers with Continuous Integration in the cloud using Hudson. With Hudson, you can spend more CPU power to help you, do more frequent build/test executions, and use static code analysis tools. Hudson is an open-source CI server that is easy to install and configure. It supports various source code repositories and build tools, records results, and notifies people. It also supports distributed builds and can automate system installations. Hudson is localized to 8 languages and has wide adoption in businesses.

bixby
Télécharger la présentation

Continuous Integration in the Cloud with Hudson

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Continuous Integrationin the Cloudwith Hudson Kohsuke Kawaguchi Jesse Glick Sun Microsystems, Inc. Hudson committers

  2. Rise of Continuous Integration • Offload from people, push to computers $ computers us time

  3. Spend more CPU power to help you • … even if it only helps a little • First on your laptops and workstations • IDEs are at the forefront • And then to the servers • a.k.a. “Continuous Integration” • More frequent build/test executions • Static code analysis tools • And more to come

  4. Hudson https://hudson.dev.java.net/ • Open-source CI server at java.net • Emphasis on ease of installation and use • “java -jar hudson.war” execution • Configure everything from browsers • Extensibility • 140+ community-developed public plugins • By 150+ contributors • Estimated 13,000 installations

  5. It basically does builds and tests • Check out the source code • Subversion, Perforce, Git, Mercurial, CVS, … • Do builds and/or tests • Ant, Maven, MSBuild, shell script, … • Record results • Binary, test results, code coverage, static analysis • Notify people • E-mail, IM, RSS, tray apps, IDEs

  6. Localized to 8 languages

  7. And hopefully more to come…

  8. Adoption in all kinds of businesses

  9. Doing Distributed Builds

  10. Why Distributed Builds? • You need to use multiple computers because… • You need different environments • You need isolation • There’s only so much you can do with 1 computer

  11. Before we talk about clouds… • Going virtual doesn’t solve… • Software installation problem • Node failure problem • Remote maintenance problem • … • What does Hudson do to help you with these?

  12. Installing new slaves • For first 20 or so slaves, we did it manually • Insert CD, click, type, click, type, click, … • But that doesn’t scale • Then we automated • Available as “Hudson PXE Plugin”

  13. Automated System Installations • Hudson + PXE plugin • ISO images of OS • Slaves • Power on, hit F12 • PC boots from network (PXE)

  14. Automated System Installations • Hudson + PXE plugin • ISO images of OS • Slaves • Power on, hit F12 • PC boots from network (PXE) • Choose OS from menu • Installs non-interactively Your corporate IT guy & his DHCP server

  15. Automated System Installations • Supports OpenSolaris, Ubuntu, CentOS, Fedora • Trivial with most Linux • Cooperate with Windows, too • Quite useful outside Hudson, too • No more broken CD drives • No more CD-Rs

  16. Distributed builds with Hudson • Master • Serves HTTP requests • Stores all important info • Slaves • 170KB single JAR • Assumed to be unreliable • Scale to at least 100 • Link • Single bi-di byte stream • No other requirements

  17. How master and slaves start talking • For Unix slaves, via SSH • Only need SSH and JRE on slaves • We just need a host name

  18. How master and slaves start talking • For Windows, DCOM • We just need admin user name and password • No manual intervention • Works even from Unix masters

  19. How master and slaves start talking • Via Java Web Start • When master cannot see slaves • A separate socket connection is made

  20. Automating JNLP launch • Once started, can be installed as Windows service

  21. Automating JNLP launch • Emulate the JNLP client headless $ java -jar slave.jar -jnlpUrl URL

  22. Automated Tool Installation - JDK • JDK from http://java.sun.com/ • Hudson automatically chooses the right bundle • Always up to date with new releases

  23. Automated Tool Installation - Apache • Ant and Maven from Apache

  24. Automated Tool Installation - Custom • Download arbitrary archive and unpack

  25. Automated Tool Installation - Custom • Run arbitrary shell commands • Can have variants by OS

  26. Automated Tool Installation - Extensible • Write your own • Simple Hudson extension • Just write Java code to create tool on slave • In progress: • SCMs – Mercurial, … • Install from Subversion

  27. Heterogeneous Cluster Challenge • Your builds/tests need to run in specific environment • Dependency on individual nodes hurts utilization jobs slaves WombatWindows test Windows #1 GlassFishWindows test Windows #2 Hudson Windows test Solaris#1 Hudson Solaris test

  28. Labels to rescue • Label is a group of slaves • Tie jobs to labels jobs slaves WombatWindows test Windows #1 Windows GlassFishWindows test Windows #2 Hudson Windows test Solaris#1 Solaris Hudson Solaris test Windows #3

  29. Forecasting failures • Hudson monitors key health metrics of slaves • Low disk space, insufficient swap • Clock out of synch • Extensible • Slaves go offline automatically • Catch problems before they break builds

  30. Clean up mess after builds • Kill runaway processes • Daemons, background processes left by your build • Works on Windows, Linux, Mac, and Solaris

  31. Load Statistics Monitoring

  32. When it’s time to add more slaves

  33. Hudson made this extensible • Hudson detects excessive workload • Hudson notifies plugins • Plugins can provision more slaves • … assuming that you have that infrastructure

  34. And that brings us to clouds…

  35. Amazon EC2: The Good • Pay as you go (10¢/h or so) • Loads on Hudson tend to be spiky • Programmable API • Instances launch at machine-speed • EC2 instances are forgetful

  36. Amazon EC2: The Bad • Your data is still inside your firewall • Takes time to check out code • … or to archive build artifacts • Some data just can’t be moved • EC2 instances are forgetful • Can your tests run in parallel?

  37. Hudson EC2 plugin • Built on top of typica* • What does it do? • Automatically provisions slaves on EC2 on demand • Picks the right AMI depending on demand • Starts slave agent • Shuts down unused instances * http://code.google.com/p/typica/

  38. Putting it all together capacity # of executors queue length usage time

  39. Hudson “Appliance” on EC2 • Run the master in the cloud too, if you like • Hudson on stock OpenSolaris AMI • Data stored persistently in Elastic Block Storage • Dynamically expandable thanks to ZFS • Online, too • Packaged as a wizard

  40. Demo

  41. Overloading your cluster

  42. Hudson Hadoop plugin • Just a few mouse clicks to install • Turn every Hudson slave into a Hadoop node • Distributed file system • Automatic data replication (fault tolerant) • Nice for storing old artifacts, logs, test records, … • Map/reduce framework • Large scale test results analysis / datamining • More interesting work to be done in the future

  43. Hudson Selenium Plugin • Selenium • Tests webapps by scripting browsers • Selenium Grid • Runs Selenium over a grid of computers • Allow Hudson labels to specify where to start browsers • Hudson & Selenium both need heterogeneous cluster

  44. Selenium Grid Hudson slaves Hudson master (selenium hub)

  45. Conclusion • CI is here to stay • We’ll continue to push more workload to servers • Hudson makes this easy for you • Reap the benefit of a cluster in multiple ways

  46. Resources • http://hudson.dev.java.net/ • BOF-5105 “Hudson Community Meet up” • Today 7:45pm same room • Hudson booth inside Sun Pavilion • Support Subscription • hudson@sun.com

  47. Kohsuke Kawaguchi Jesse Glick kohsuke.kawaguchi@sun.com jesse.glick@sun.com http://hudson.dev.java.net/

  48. Agenda • Quick Hudson introduction • Doing distributed builds • How Hudson make distributed builds easier • Taking infrastructure to cloud

  49. Setting up slaves • Keeping slaves consistent is a good thing • Particularly hard on heterogeneous environment • General system administration tasks • Network configuration • Package installations for native tools • Tools like Puppet or cfEngine are supposed to help • Install build tools in the cluster • Prepare tools on one file system • rsync to everywhere • This part of Hudson needs improvements

  50. Hudson EC2 plugin usage • Tell Hudson your AWS account information

More Related