1 / 24

What’s New in Condor

What’s New in Condor. Overview. Quick ‘sound bytes’ on new functionality in recent Condor releases Condor Development Process New Features in Condor version 6.6.x New Features in Condor version 6.7.0. Condor Development Process. We maintain two different releases at all times Stable Series

chase
Télécharger la présentation

What’s New in Condor

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. What’s New in Condor

  2. Overview Quick ‘sound bytes’ on new functionality in recent Condor releases • Condor Development Process • New Features in Condor version 6.6.x • New Features in Condor version 6.7.0

  3. Condor Development Process • We maintain two different releases at all times • Stable Series • Second digit is even: e.g. 6.2.2, 6.4.7, 6.6.3 • Development Series • Second digit is odd: e.g. 6.5.1, 6.7.2

  4. Stable Series • Heavily tested • Runs on our department production pool of nearly 1,000 CPUs (for min of 3 weeks) • No new features, only bugfixes and ports. • A given stable release is alwayscompatible with other releases fromthe same series • 6.6.X is compatible with 6.6.Y • Recommended for production pools

  5. Development Series • Less heavily tested • Runs on our small(er) test pool. • New features and new technology areadded frequently • Versions from the same developmentseries are not guaranteed compatible with each other (although we try hard)

  6. New in version 6.6.x • Version 6.6.0 released in November 03. • Current release: version 6.6.3, released in April 04.

  7. The Struggle to Build Condor • Condor is BIG • Condor code consists of primary source plus ‘externals’. • Externals include Kerberos, zlib, GSI, PVM, gSOAP… • Patches to externals

  8. The Struggle to Build Condor • Condor is BIG • Condor code consists of primary source plus ‘externals’. • Externals include Kerberos, zlib, GSI, PVM, gSOAP… • Patches to externals • Current shipped source + externals: ~415MB of source, or ~9 million lines! • Building Condor outside of UW-Madison used to be very difficult. • “LIST OF SHAME”: Build pointed to packages on UW-Madison fileservers.

  9. Now Condor Source “Self-Contained” • Source code to externals are now bundled w/ Condor itself. • Self-contained • Allows version control on externals + patches • Build w/ just “configure; make” ! • Checks for existence and proper version of all “bootstrap” requirements, such as the compiler • Applies our patches to the externals • All 9 million lines built and bundled

  10. Building Condor Building Condor before Version 6.6.0… Building Condor Post Version 6.6.0!

  11. Condor + NMI • NMI = NSF Middleware Initiative • Automated build and test infrastructure built on top of Condor • Pool of 37 machines of many architectures • Scalable • Runs every night, builds several Condor source branches, then runs 114 test programs. • All results stored in RDBMS, reported on the web. • Yes, Condor builds Condor!

  12. Ports • New Ports w/ v6.6.x –vs- v6.4.x : • Solaris 9 • RedHat Linux 8.x, 9.x for x86 (+RPMs) • RedHat Linux 7.x and SUSE 8.0 for IA64 (clipped) • Tru64 5.1 (clipped) • AIX 5.2 (clipped) • Mac OS X (clipped)

  13. Some new components • Computing On Demand (COD) • Integration of “Hawkeye” technology • Condor-G Additions • Matchmaking • Grid Monitor • Grid Shell

  14. Computing On Demand (COD) • Introduce effective timesharing to a distributed system • Batch applications often want sustained throughput for a long period of time • Interactive applications often want a quick burst of CPU power for small period of time • COD : Allow both to co-exist

  15. HawkEye Technology • Dynamic Resource Monitoring, now ‘built-in’ to Condor. • Allows custom dynamic attributes to be added into machine classads. • These attributes can be used for • Queries • Scheduling • Many plugins available. • Disk space, memory used, network errors, open files/descriptors, process monitoring, users, …

  16. Condor-G • Condor-G Matchmaking • Condor-G can determine which grid site to utilize via ClassAd matchmaking (grid planning, meta scheduling, …) • Condor-G Grid Monitor • Reduces the load on a GT2-based gatekeeper, greatly increasing the amount of jobs that can be submitted • Condor-G GridShell • A wrapper for the job • Reports exit status, cpu utilization, more

  17. Improvements in Condor for Windows • Ability to run SCHEDULER universe jobs • Including DAGMan • JAVA universe support • More Win32 flavors, incl international versions. • Added support for encryption on disk of the job and data files on execute machine.

  18. New Features in DAGMan • DAGMan previously required that all jobs in a DAG share one log file • Each job can now have it’s own log file • Understands XML formatted logs • Can draw a graphical representation of your DAG • Uses GraphViz, http://www.graphviz.org/

  19. Central Manager New Features • Central Manager daemons can now run on any port COLLECTOR_HOST = condor.cs.wisc.edu:9019 NEGOTIATOR_HOST = condor.cs.wisc.edu:9020 • Useful for firewall situations • Allows multiple instances on one machine • Keeps statistics on missed updates • Can use TCP instead of UDP, if you must

  20. Command-line Tools • ‘condor_update_stats’ tool to display information on any dropped central manager updates • ‘condor_q –hold’ gives you a list of held jobs and the reason they were put on hold • ‘condor_config_val –v’ tells you where (file and line number) an attribute is defined • ‘condor_fetch_log’ will grab a log file from a remote machine: • condor_fetch_log c2-15.cs.wisc.edu STARTD • ‘condor_configure’ will install Condor via simple command-line switches, no questions asked • ‘condor_vacate_job’ to release a resource by job id, and can be invoked by the job owner. • `condor_wait’ blocks until a job or set of jobs completes

  21. New 6.7.x Development Series • Release of v6.7.0 in April 04. • Can you take the suspense?!?

  22. V6.7 Themes • Scalability • Resources, jobs, matchmaking framework • Accessibility • APIs, more Grid middleware, network • Availability • Failover

  23. Thank You! • Later this afternoon is the roadmap for future work. • Questions?

More Related