CMS Components Generic Infrastructure Bits

1. CMS ComponentsGeneric Infrastructure Bits Lassi A. Tuura Northeastern University, Boston Lassi A. Tuura, Northeastern University

2. Frameworks: Structure Overview Graphics Classlib Plug-in Manager Components Consistent User Interface Data Browser Generic analysis Tools GRID Distributed Data Store & Computing Infrastructure Analysis job wizards Objy tools ORCA COBRA OSCAR FAMOS Detector/Event Display CMS tools Federation wizards Coherent basic tools and mechanisms

3. Frameworks: Layering ReconstructionAlgorithms EventFilter PhysicsAnalysis DataMonitoring (Grid-aware) Data-Products Overview Graphics Classlib Plug-in Manager Components Specific Frameworks Grid-Uploadable Physics modules Calibration Objects Generic Application Framework Configuration Objects Event Objects Adapters and Extensions ODBMS Basic Services GEANT 3 / 4 CLHEP PAW Replacement C++ Standard Library + Extension Toolkits

4. Generic Components (Grid-aware) Data-Products Overview Graphics Classlib Plug-in Manager Components Specific Frameworks Grid-Uploadable Physics modules ReconstructionAlgorithms EventFilter PhysicsAnalysis DataMonitoring Calibration Objects Generic Application Framework Configuration Objects Event Objects Adapters and Extensions GEANT3 / 4 Basic Services Qt Classlib Open Inventor C++ Standard Library + Extension Toolkits

5. Quick IGUANA Overview Overview Graphics Classlib Plug-in Manager Components • A generic toolkit for user interfaces and visualisation • Builds on existing high-quality libraries (Qt, OpenInventor, Anaphe, …) • Used to implement specific visualisation applications in other projects • Used in CMS and D0 (and L3—pretty static) • Portable: UNIX and Windows actively used • Few limits to porting to new platforms • Main technical focus: provide a platform that makes it easy to integrate GUIs as a coherent whole, to provide application services and to visualise any application object • Many categories / layers: GUI gadgets & support, application environment, data visualisers, data representation methods, control panels, … • Designed to integrate with and into other applications • Apart from a tiny kernel, virtually everything else is in plug-ins • Plug-ins and dynamic linking are 100% orthogonal

6. Quick IGUANA Overview Overview Graphics Classlib Plug-in Manager Components Integration Action Visualise Results, Modify Objects, Further Interaction

7. Building Blocks Overview Graphics Classlib Plug-in Manager Components • Basic services layer • Portability + utilities Classlib • GUI and 2D graphics Qt • 3D graphics OpenInventor, OpenGL • Internal utility services • Plug-in manager • Generic base • Concrete implementation • (More details later on) • Internal state management and service components • (Components, abstract interfaces and concrete classes)

8. Graphics Toolkits Overview Graphics Classlib Plug-in Manager Components • GUI and 2D Graphics: Qt • Easy to use • Very rich functionality, lots of community experience • Very portable, uses platform GUI conventions: looks “natural” • Excellent documentation • Free for our purposes • 3D Graphics: OpenInventor • Same as above—portability from OpenGL; both commercial and free exist • An incredibly rich toolkit! • Qt and OIV limited strictly to these parts of IGUANA • Not using Qt or OIV classes (containers and many other utilities) outside their respective “natural” graphics domains: other parts use other libraries • Reasons explained when discussing our component approach

9. Debugging + Robustness Overview Graphics Classlib Plug-in Manager Components • Debug Assertions; Validation; Stack tracing; Programmatic core dump • Log Rich debugging-oriented logging facility • Signal POSIX; Standard C/C++; Win32 SEH; Fatal error handling and recovery • LowMemoryPool Graceful low-memory protection • Error Exception chaining for increased error abstraction as errors are propagated through the application (much used in implementation) • SaveErrno

10. Major Tools Overview Graphics Classlib Plug-in Manager Components • xtype_info • Extensible std::type_info replacement • Minimal meta-class information • MultiMethod • Multi-method dispatcher • Works with arbitrary argument lists • “Arbitrary”: Any polymorphic class can be used as “virtual” argument;any number of “virtual” and non-virtual arguments can be dispatched • Macro sugar for up to 3 virtual arguments and 2 non-virtual args • Optimised to be storage and performance efficient even if hundreds of classes and methods exist • Registry • Sort of like Windows registry but concept modified to actually work

11. System Abstraction Overview Graphics Classlib Plug-in Manager Components • SharedLibraryShared library loading, unloading; list of currently loaded libraries • Time, TimeSpans-resolution calendar time, conversions, formatting, calculations • TimeInfoHigh-resolution (ns) monotonic time, system and process times • HostInfoHost itself, DNS lookups • ProcessInfoPID etc • ResourceInfoProgram + system resource info, limits • SystemInfoSystem information: memory, os, … • UserInfoUsers: home dir, login, real name, …

12. Bits & Numerics Overview Graphics Classlib Plug-in Manager Components • BitIteratorIterating over data as bits • BitOpsCollection of optimised bit magic operations • BitPatternConstruction of bit patterns • BitTraitsBit type properties of integral types • IntBitsType selection by required number of bits: {exact, fast} x {signed, unsigned} • IntTraitsPoor man's std::numeric_limits

13. Miscellaneous Overview Graphics Classlib Plug-in Manager Components • Environment, ShellEnvironment • Callback, Hook, HookList • MapIterator • PODVector • Range • Sequence • SearchPath • Unwind • UUID • Utility Macros

14. I/O: Basics Overview Graphics Classlib Plug-in Manager Components • IOChannelAbstraction of a non-seekable I/O sources and targets • StorageAbstraction of a seekable (file-like) storage • IOBuffer, IOSize, Basic I/O insulation: systemIOOffset characteristics (e.g. LFS) • DirIteratorIterator for reading directory contents like readdir()

15. I/O: Files, File Systems Overview Graphics Classlib Plug-in Manager Components • File, FileAcl, FileMapping • File operations • File and anonymous memory mapping • High-throughput I/O channel transfers (e.g. file to socket) • Filename • Filename abstraction • Existence, type, permissions checking • Core support for file backup, versioning, autosaving, locking • Filename expansion, substitution, abbreviation, completion, globbing • File rename, move, copy • Secure temporary file handling • Current working directory • Description of system file name conventions

16. I/O: IPC Overview Graphics Classlib Plug-in Manager Components • IOChannelPairPair of I/O channels (e.g. pipe or socket pair) • SocketAll socket stuff (TCP, UDP, UNIX, …) • InetSocket, InetServerSocket, InetAddress • LocalSocket, LocalServerSocket, LocalAddress • SocketPair • NamedPipeFile system FIFOs • Pipe Anonymous FIFOs • PipeCmd “popen()” • IOSelectorI/O switch a la “select” (implementations available: select, poll, /dev/poll, real-time signals, sigio; Win32 select simulation, overlapped I/O (completion ports?))

17. I/O: High-Level Streams Overview Graphics Classlib Plug-in Manager Components • Input/Output Streams • Base interface • Buffered • Filtering • Linked • Pushback (input only) • C++ Standard (std::i/ostream) • C Standard (FILE *) • Storage (cf. I/O Basics) • IOChannel (cf. I/O Basics) • Storage implementations • Memory • C++ Standard • C Standard

18. Compression + Archives Overview Graphics Classlib Plug-in Manager Components • Data checksums • Interface, Adler32, CRC32 (MD4, MD5?) • Compressor and decompressor engines • Interface, BZIP, ZLIB • Generic interactive engine • Input & output streams • Compressing and decompressing base interface • BZIP, ZLIB, GZIP, ZIP, TAR • Checksum • Archives files and their members • TAR, ZIP

19. Modules Overview Graphics Classlib Plug-in Manager Components • What’s a module? • An (extension) component to a framework (often dynamically loaded) • Framework knows only that a module extends a particular API (usually abstract interfaces), not how and what it needs to do so • Examples abound: Netscape plug-ins, Apache modules, Python and PERL modules, Illustrator and PhotoShop extensions, … • Already used in many ways in CMS projects: package builders, detector units, hit/digi readers and writers, and many algorithms are essentially extensions of APIs provided by COBRA, and “subframeworks” in ORCA • Why modules? • Eases management—framework doesn’t need to know lots of details about the modules, but can simply load them and gain additional capabilities • Helps to give symbolic names to capabilities and allow framework to reason about them: “here’s all you will need to read data product X” • Allows better services to external parties (Python, GRID, IGUANA, …) without having to put that knowledge into many applications

20. The Plug-In Manager Overview Graphics Classlib Plug-in Manager Components • Maintains a list of directories with plug-in definitions • Plug-in = shared library with well-defined entry points • Self-describing through a query entry point • But really orthogonal to dynamic loading • Some or all plug-ins can be built into the program • Automatically detects new, changed or removed plug-ins • For new or changed ones loads the plug-in, queries for its capabilities, and unloads the plug-in • Maintains a cache of plug-in properties • Updates the cache if the directory is writable • When requesting information from the database, uses the cache instead of loading all the plug-ins all the time • A separate generic package • IGUANA architecture defines a specific implementation

21. Overview Graphics Classlib Plug-in Manager Components

22. Abstract or Concrete? Overview Graphics Classlib Plug-in Manager Components • Abstract or concrete implementations? • We use both: a lot of abstract interfaces and decoupling at the bottom level,more concrete implementation usage on the top • If you need a specific service, you use it directly • Access through shared state object which records references to “application state extensions”, but knows nothing about them • Clients ask specific services to look themselves up in the state • Services vary in “abstractness”: current selection state does not need to know anything about GUI, object context menu is defined in terms of Qt • Only if the service has multiple alternative implementations the service has an abstract interface; otherwise clients use a concrete implementation • Services can and do have mini-plug-in architectures of their own • Proxies and envelopes sometimes better than abstract interfaces • Catch-all IgExtension for those mini-plug-in systems but it’s not an IInterface • Concrete plug-in manager defines the interfaces specific to its mission • Trying to keep things simple and straightforward!

23. External Components Overview Graphics Classlib Plug-in Manager Components • Qt and OIV: use concrete implementations only • Not trying to encapsulate these in any way • … but restrict them to their “native domains” • Reason: build modularity firewalls in the system • No point in trying to abstract so rich toolkits • Major benefits in going for full integration • Changing package known to incur limited pain • We know exactly what we are using and from where • We know we can get rid of the original package • Plan: expend N months to swap with equally rich or better product