1 / 11

Red Hat on Jenkins: By the numbers

Red Hat on Jenkins: By the numbers. Nam Duong Red Hat, Inc. https://openshift.redhat.com. Agenda. Products currently leveraging Jenkins Deployment size: # of Masters/Slaves/Jobs in use across Jboss Middleware QE Top 10 list of interesting Plugins Code contributions

venus
Télécharger la présentation

Red Hat on Jenkins: By the numbers

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Red Hat on Jenkins: By the numbers Nam Duong Red Hat, Inc. https://openshift.redhat.com

  2. Agenda • Products currently leveraging Jenkins • Deployment size: # of Masters/Slaves/Jobs in use across Jboss Middleware QE • Top 10 list of interesting Plugins • Code contributions • Experiences/Pitfalls/Lessons learned • Best Practices • Demo: • Production/Staging server (sheer numbers) • OpenShift on Jenkins • EC2 and GIT plugin demo

  3. Red Hat products • Products currently leveraging Jenkins • JBoss offerings: • Enterprise Application Platform • JBoss Developer Studio • SOA • JBoss Portal • JBoss OSGI • Many, many others • OpenShift – OpenShift is our Platform as a Service (PaaS) offering. Our Jenkins instances are running in the cloud, hosted on OpenShift Flex and used to build and test OpenShift

  4. Deployment Size • JBoss Middleware Team: • 1 Master • IBM System x3550 M3, 2x E5640 CPU, 24GB RAM, 2x SAS 73GB 15K-RPM in RAID1 • Slaves connected 113 (various OS: linux, Win, Solaris, HP, AIX) • ~4700 Jobs configured (~1000 running/month) • ~95% of Middleware test automation is called/executed  via Jenkins • OpenShift Flex Team: • 1 Master • 7.5 GB, 4CPU, 10GB EBS (RHEL 5.6) • 10 Slaves Building EC2 Images and 5 slaves running tests • 20 jobs (RHEL 6.1) • ~90% of our automation is executed via Jenkins

  5. Interesting plug-ins used • Management (Groovy*, Disk Usage*, Build Publisher*, Workspace Cleanup*) • Cloud Control (EC2 plugin,  DeltaCloud*) • Enterprise focused plugins (SmartFrog) • Code Coverage (cobertura, emma) • Source control (GIT/SVN/CVS) *Represents code contributions

  6. Code Contributions • Memory leaks:  Involved in diagnosis, resolution • In the Hudson timeframe, we contributed to stability through reporting of bugs/instability (large deployment) • First ones to run into these types of problems (pure volume) • LTS (long term support - Jenkins stable branch) • Development and maintenance of plugins (see previous) • reporting bug + bug fixes

  7. Experiences/Pitfalls/Lessons learned • 1 master may not handle all jobs at once - potentially provide failover functionality on Jenkins • Jobs getting stuck due to polling console output through ssh. Need to limit jobs from writing to console. • Disk Utilization issues - Overcome through quotas set on jobs • Logging was too verbose, Jenkins instance stalled due to low/no disk space • OOM issues when trying to view the giant logs - fix in progress

  8. Best Practices • Set limits on memory usage (quota) of slave • Set limits (quota) on Disk usage for jobs • Tools on shared NFS drive – for maintenance and replication/recovery • Memory – lots of memory to buffer incremental growth of Master • Jenkins master and all slaves under CFEngine control (config management/recovery) • Like all reliable services, use Dev/Stg/Prod increments. • Can scale vertically, but has anyone tried scaling horizontally?

  9. Demo • Production instance within corporate firewall: • Production: <Screenshots available upon request> • Staging our plugins • Staging: <Screenshots available upon request> • Jenkins running on OpenShift Flex; testing OpenShift Flex • <Screenshots available upon request> EC2 and GIT plugin demo • Used to create a client testbed • Creates an EC2 instance, prints the IP address, pulls down all code bits

  10. Links/Contact • Email: nduong@redhat.com • https://openshift.redhat.com • We appreciate your feedback of your user experience

  11. Thank You To Our Sponsors • Coming Soon: The CloudBees Newsletter for Jenkins • Please complete the Jenkins survey to help us better serve the community • (bonus: a chance to win an Apple TV!)

More Related