Scalable Performance Visualization with Jumpshot Omer Zaki, Rusty Lusk, Bill Gropp, Debbie Swider Mathematics and Computer Science Division Argonne National Laboratory
Not Included • Getting performance with MPI • applications • implementation research topics • MPI-2 • contents • implementation availability • MPICH • The MPI-2 approach to parallel I/O
Outline • What is Jumpshot and where did it come from? • Related efforts • Our requirements for a logfile-based performance visualization system. • Producing logfiles: CLOG • Visualizing logfiles: Jumpshot • Java issues • Future work
What is Jumpshot? • Tool for understanding the behavior of parallel programs • Post-mortem • Logfile-based • Includes logging package (CLOG) • Primarily for MPI programs • Written in Java processes CLOG logfile Jumpshot display
Typical Jumpshot Screen
Instrumented version of PETSc
Where Did Jumpshot Come From? • The history of logfile-based performance-analysis tools at Argonne is also the history of the search for a programming environment in which to implement simple graphics plus a GUI. • Gist (BBN) -- raw X • black and white, not portable (BBN Butterfly only) • Upshot -- raw X plus Athena widgets, used with ALOG logfile format • painful, especially to get performance (1990) • Upshot redone in Tcl/Tk • easy to write, but graphics too slow
History (continued) • Nupshot -- Upshot redone in Tcl/Tk/C for speed • good performance • Tcl/C interface unstable • CLOG -- new log format for many reasons • Java + Upshot = Jumpshot • uses CLOG • has new features • explores Java technology • next up: JPython?
Related Efforts • Gist survives in Dolphin’s TotalView as TimeScan • PICL/ParaGraph - Pat Worley, Mike Heath, Jennifer Etheridge, Al Geist • VAMPIR - Pallas • Traceview - Al Malony • Pablo - Dan Reed, Ruth Aydt • XPVM - Jim Kohl, Al Geist • XMPI - Raja Daoud, now Notre Dame • Paradyne - Bart Miller, Myron Livny • others
Why Do It Again? Requirements for a new system (not all of them yet satisfied by Jumpshot): • stable environment for long lifetime to accommodate future research • portable, even unto Microsoft • support for upshot-type views that we have found most useful • process timelines with scrolling and zooming • histograms of state durations, message properties, mountain ranges • animation not so useful
Requirements (continued) • flexible, extensible logfile format to accommodate new types of events, states, concepts • end-user-defined states • scalable performance • control of logging at source • aggregation • at least tens of thousands of events • nested and overlapping states • nested more important than overlapping • connection of displayed events back to source code.
Requirements (continued) • MPI awareness (communicators, semi-transparency of collective operations) • ability to query details of specific events, messages, and states. • ability to locate “interesting” parts of display (research topic) • a new one every week....
The CLOG Logging Library: Background Characteristics of old ALOG: • fixed-format records (6 ints and short string) • good for parsing, storing, access • bad for extendibility • timestamps an integer number of microseconds • OK in 1990 • not accurate enough now • ASCII format in file • good for portability, easy to read • can’t store binary data conveniently
CLOG: Requirements • Efficient enough to not interfere with behavior of program; I/O only when program finishes • Only one logfile at end is convenient • Timestamps not assumed synchronized • Flexibility in record type, but not completely self-describing • Portable: logfiles can be read on different machine than one where written.
CLOG: How It Works • relies on MPI, for portability • calls MPI_Wtime to get timestamps • reasonably, but not ultimately, efficient on any given architecture/OS • multiple record formats with types, plus “raw” type • user can define own types, states, colors • log records accumulate in big buffers in memory until malloc fails, then stop
How CLOG Works (cont.) • At end (CLOG_Finalize is collective) • process-local data is added to buffers • timestamps are adjusted, using simple algorithm and communication with other processes • processes form a binary tree and do local 3-way merge in parallel up to process 0, which writes file • file is in Java (MPI-2’s “external-32”) format
“Normal” Jumpshot Features • Scrolling and zooming in timeline view • Arrows to represent messages • Click on arrows and states for details • Histograms of state durations, message bandwidth • Mountain range view to aggregate state info • Can select/deselect states, messages
Timelines and Mountain Range
“Unusual” Jumpshot Features • Scrolling and zooming in histogram view • Can focus on extreme durations/bandwidths • calculate top/bottom 1%, 5%, ... based on assumed normal distribution • blink corresponding state instances, arrows; can help locate “outlier” events in large confusing display. • Can scroll timelines individually to fine-tune clock synchronization • Inherited from Java: • portability • can be run as applet • fancy GUI features • multiple look-and-feel • tear-off subwindows
Java Issues - Good • Portable (Sun, SGI, RS6000, Windows, NT) • Can be run either as normal X application or with a browser as an applet • Graphics are fast enough • Widget set is more than adequate for GUI construction - Swing.
Java Issues - Awkward • Java still rapidly evolving in this area (1.0 - 1.1.2 - 1.1.6 - 1.2beta) • made many big changes during two-month period to deal with bugs, re-implemented features • add-ons also evolving (e.g. swing) • applet behavior not completely consistent with application behavior • Inconveniences arising from the fact that a Java program is not self-contained (e.g., CLASSPATH environment variable) • partially resolved with use of JRE • We are still committed.
Jumpshot Distribution • Jumpshot currently comes with • CLOG library for creating logfiles, uses any MPI • mpe logging library with ALOG/CLOG switch • MPI profiling library for automatic instrumentation of MPI programs • Is distributed • as part of MPICH distribution (version 1.1.1) • separately as part of mpe library, for use with any MPI implementation • separate jumpshot only
Future Work • Meet more of the requirements • connection to code (by logging __FILE__, __LINE__) • select by MPI communicator (not so easy, because of communicator id issue) • vertical scrolling • Research on scalability issues; useful agglomerations of data • Research in automated detection of performance anomalies
The End
