RPC PAC trigger software

1. RPC PAC trigger software CMS L1 Trigger Online Software Review, 5February 2009

2. To TS Central Cell To RCMS Top FM RPCT TS Cell RPC FM TS Subcell TS Subcell TS Subcell … LBox access XDAQ LBox access XDAQ DCC access XDAQ … Monitoring and test manager JAVA TC accessXDAQ SC accessXDAQ … VME DCC/CCS crate VME VME FEC FEC DCC … Trigger and Sorter Crates CCU rings LBox’es … DB service TStore I2C rings Condition DB Config DB FEB FEB Dashed = planned FEB FEB … RPC PACT online software schema Karol Buńkowski, Warsaw University

3. Hardware Access XDAQ • Hardware Access (HA) XDAQs • The only applications with direct access to the hardware via the VME interface, • One application per each Trigger Crate (12TC+1SC); XdaqTestBench class, • One application per each CCU ring (= 6 LBoxes= 90 LBs), in total 18 applications; XdaqLBoxAccess class, • Functionalities: • Hardware setup: • Retrieve hardware structure from DB and build objects corresponding to devices, • Retrieve configuration data (including firmware version ID) for the given config key from the DB, • Check if correct firmware is loaded, reload if necessary, • Check the hardware status (TTCrx, QPLL), reset if necessary, • Check if correct config data are applied (ID of config data version is written in a dedicated registers), if not – reconfigure. • Hardware monitoring, • Access point to the hardware for other applications via the SOAP (e.g. FEB config XDAQ, JAVA Test Manager). Both low level functions (register read/write) and high level (massive configuration of FEBs or diagnostic modules) available via SOAP, as well as info about hardware structure. 5 computers used to control the hardware, on them the HA XDAQ are running.The XDAQs are running as services, but started manually (what is need at the moment for debuging). Starting through the CMS Trigger-Process Controller can be configured at any time. Karol Buńkowski, Warsaw University

4. HA XDAQ GUI - TC In standalone mode: available config keys from DB. If controlled by TS – the key is received from TS Configure options Monitoring control Hardware Status monitorables Monitoring histograms Karol Buńkowski, Warsaw University

5. HA XDAQ GUI – Link System Firmware upload Monitoring: Chambers rate Karol Buńkowski, Warsaw University

6. Trigger Supervisor • Present RPC TS architecture: RPC TS central cell 13 TS subcells(one for each TC and SC)  13 standalone TC/SC HA XDAQs • Each subcell communicates always with one XDAQ to allow parallel configuration using TS framework. • Functionalities of the central cell: • MTCCIIConfiguration operation to configure the system: very simple procedure: pass configuration command with configuration key to HA XDAQs via the subcells. • Monitoring: collects, analyses and displays the monitoring data obtained as flash-lists and monitoring charts (png files) from the HA XDAQs All cells are started by CMS Trigger-Process Controller Karol Buńkowski, Warsaw University

7. Databases • Configuration DB (~ 60 tables) • Hardware structure • Chambers structure: location (no geometry info), strips, FEBs, connections, • Electronic hardware components (crates, boards, chips) and their relations and connections , • Assignment of hardware partition for given HA XDAQ, • List of disabled devices • Configuration data: config key, register values, firmware ID, name of the script for FPGA loading. • Fully implemented and used • Condition DB • Not implemented yet • Planed to put the monitoring flash list to the DB with the TStore • Not clear what to do with the monitoring data from the histograms, which are currently stored in the ROOT files. Karol Buńkowski, Warsaw University

8. DB Service JAVA, Created in the similar technology as the Function Manager. • Tomcat (simple JAVA www server) is used to deploy DB Service as a web application. • Access to DB Service is provided by SOAP using Apache AXIS library. Implemented custom serializers/deserializers for Axis so that integration with XDAQ is very simple. • Access to Oracle through JDBC and Hibernate. Java package (rpct.db.domain) with object representation of all database entities: hardware structure and configuration data. This package is also used by Central Test Manager (JAVA), what allows fast and organized access to the configuration database. Karol Buńkowski, Warsaw University

9. Central (Monitoring and)Test Manager • JAVA software, which communicates with the HA XDAQs via SOAP (Axis), direct access to configuration DB using rpct.db.domain package (JDBC and Hibernate). • Architecture: • Objects corresponding to the devices are crated based on the DB; interconnections are also represented, • Second set of objects corresponding to the devices and diagnostic modules are crated based on the info from theHAXDAQs, provide the hardware control via SAOP and HA XDAQs, • Mapping between above object into HW-DB objects, • TestManager for performing test with diagnostic readouts and test pulse generators, • EventBuilder for analysis of data from diagnostic readouts (base of the interconnection test), • Mapping of the config data from the DB to the JAVA objects – used for putting the config data to the DB. • Allows to execute complex processes from one application for whole system (e.g. configuration of the diagnostic modules), • Main applications: • Interconnection tests, • Algorithm implementation tests, • Finding hardware configuration parameters (e.g. synchronization of the transmissions), the parameters are directly uploaded to the config DB, • dedicated applications for hardware debugging and testing (easy to write, fast development), • Planned to use for complex monitoring tasks, which required to talk to many different HA XDAQ (e.g. LB-TB transmission monitoring). • Currently run as command line applications but first successful attempts has been made to deploy them as a web service what allowed to execute the tests by the Trigger Supervisor, Benefits from JAVA easiness of software development and debugging (Eclipse graphic debugger) and advanced libraries for DB, XML, etc. Karol Buńkowski, Warsaw University

10. InternalInterface Access – JAVA web application • Functionalities • Application for hardware debugging • devices tree: crates, boards, chips, registers • Reading and writing of each register • Architecture • Tomcat as Java WWW server (servlet container) • Axis with custom serializers/deserializers to make communication with XDAQ's easy • Takes info about hardware from HA XDAQs by SOAP • register read and write via XDAQ • JavaServer Faces web framework to build rich internet applications (e.g. we use powerfull  AJAX TreeTable) Karol Buńkowski, Warsaw University

11. Hardware monitoring Monitoring is performed directly by the HA XDAQs • Monitoring of the hardware status– status monitorables: • Device status (e.g. TTCrx ready, QPLL locked) – simple check, if not ok– ERROR is issued, • Transmission error counters: WARNIG or ERROR is issued depending on the error rate, The process in XDAQ reads repetitively the values of selected registers: • separate thread with timer, sampling rate: every few seconds. One board is read per one timer tick, in this way the HA XDAQ is not blocked for long time (important in case of the Link System, where many board in one HA XDAQ). • The results are printed on the XDAQ web page, and are put to the flash list. • Flash list contains OK status for each board or WARNING or ERROR with description (e.g. rate of the error counters). • Monitoring of the trigger data (non-event data) – statistic monitorables (counters and histograms analyzing the data stream):Separate thread with timer: • the diagnostic modules are configured, • the diagnostic modules are started for 10 sec. • the data from the diagnostic modules are readout and analyzed, the plots (accumulated histograms, last histograms, history plots) are created with use of ROOT and displayed by the XDAQ web page, • The “raw” data of every iteration are saved in the root file, so that other analysis can be performed offline. Karol Buńkowski, Warsaw University

12. TS snapshot I – flashlist display TS central cell collects, analyses and displays the monitoring data obtained as flash-lists and monitoring charts (png files) from the HA XDAQs Click here to get list of monitorables Overall status of each TC Karol Buńkowski, Warsaw University

13. TS snapshot II - Output rate history Karol Buńkowski, Warsaw University

14. Propagation of configuration data to offline From config DB • Cabling map (from chamber strips to PAC inputs) – mature PopCon application, reads several OMDS tables and fills out RPCEMap object. Frequent usage is not required – manual operation when needed is fully sufficient (agreed with DB group), no upgrades foreseen. • PAC patterns- Up to recently a dedicated PopCon application has been used – will be superseded, now integrated with L1 O2O infrastructure.Full sets of patterns are predefined well before the start of run and given in form of xml files. Preloading jobs (O2O – based) to transfer xml files to DB and associate payloads to configuration keys - exist and have been tested. • Logical cones (PAC inputs definition) – constructed dynamically from ascii files. Creation of a dedicated DB object which contains those parameters is in progress. Plan is to have it tested before the deadline for CMSSW_31X. Should be integrated with L1 O2O. Karol Buńkowski, Warsaw University

15. Development of missing items, consolidation plans and priorities • HA XDAQ: • in principle complete – almost all major functionalities implemented, but some polishing and debugging is needed. • If manpower available we would like to convert the HA XDAQs to TS sub-cells, • Upgrade to XDAQ7. • TS: • TS central cell almost ready. In practice TS was not used for configuration due to the need of running the opto synchronization software procedure as a part of config process (delay of ~15 mins). Configuration so far has been done by hand with use of HA XDAQs. With new firmware that performs automatic optical links synchronization, we can start configuring via TS. Short term task: • Test the integration with the HA XDAQs, • Integrate LBOX XDAQs with TS for configuring - make dedicated TS sub-cell for them, • Integrate LBOX XDAQs for monitoring – straightforward at TS level, LBOX XDAQs must send their own flashlists with hardware statuses, • Upgrade to TS 1.6 Medium term tasks: • Make use of sentinel error messages and alarms • Other tasks: • Improve FEB configuration (speed up) – create the software and test the firmware of improved I2C controller • Finish implementation of the software and firmware features for automatic LB reloading and reconfiguration, Karol Buńkowski, Warsaw University

16. Software repository and documentation • Organization and management of the software repository: • Currently CVS repository on the server in Warsaw • We want to move the CERN repository, but it needs some effort… • Status of documentation: • TWiki page with users manuals (for experts - Warsaw server, for others – CMS TWiki), • Documentation of sources – only some comments in not straightforward sections of codes, Karol Buńkowski, Warsaw University

17. Manpower • Main developer of the online software - Michał Pietrusiński. However presently working with us part-time (15%), cannot spend much time on CMS. • Karol Buńkowski: HA XDAQ, JAVA, • Mikołaj Ćwiok: LinkBoxAccessXDAQ • Michał Szleper: O2O Karol Buńkowski, Warsaw University

18. Questions and answers • Which programming languages are you using? : C++ and JAVA • Which external frameworks/middleware/packages are you using for the online software? : Axis, Hibernate, Maven, Tomcat, • Can you make a build of all the online software in one step? :C++ in one step, a few steps for JAVA, • Do you have automatic tests for the Configuration operation? : No • Do you have a test to assure the hardware functionality in one step? : Hardware setup and monitoring provides (almost) full tests of hardware functionality, • How are you implementing the Interconnection Tests? Is it a one step process? : JAVA Central Test Manager; a few standalone applications for a few types of tests, • Are you using the HAL library to access to the hardware? : No Are you accessing to the Linux drivers directly? : Yes • Are you using C++ namespaces? : Yes rpct • Do you have a hierarchy of exceptions inherited from CellException to describe all the possible malfunctions of the software? : No • Do you have cout or cerr in any part of the online subsystem code? : some debug (which should be cleaned) • What is the worst case configuration duration? : a few minutes in case of the firmware loading • What is the worst case duration of a control panel event? A few seconds (in XDAQ standalone mode) • Are you using global variables in the subsystem online software? : No • How much memory is leaked in each sequence of: create configuration operation, configure, destroy operation? : Not tested recently, not problems experienced Karol Buńkowski, Warsaw University