Configuration, service instrumentation & service management

1. Configuration,service instrumentation & service management Joachim Flammer on behalf of the JRA1 Integration team

2. Overview • Comments • Requirements • Service interactions • Common issues • Implementation scenarios • DM JMX example • Discussion points

3. A few comments to start … • Timeline • Discussion in the area of configuration an service management is going on for several months already • If we want to have something in gLite 1.x we don’t have much time left • Framework • So far discussion concentrated much on configuration, dynamic configuration, … • … however the scope is much wider and is about proper service management and instrumentation • Priorities • In our opinion a proper working service management is a key corner stone of gLite • It will up to the project to decide for the priorities • Discussion started in PTF … • … but we need to have a proposal ready

4. Requirements • Do we need service instrumentation ? • There has been a lot of discussion … … clear project made statement about the requirement • JRA1 / SA1 presentation in Athens • Discussions with SA1 • Discussions with user community • PTF Requirements Database

5. PTF Requirements Database • Many entries related to configuration / management / logging & error handling • Entries are coming from different sides (SA1, applications, …) • Some examples: • 100724 • Grid services should have an administrative interface for: • monitoring • creating alarms • taking out sites/services by grid operation center etc... • They should have common, extensible, set of APIs • 100727 • All grid services must be able to use standardized interface for error and log messages and files. Log level has to be adaptable for e.g. different level of debugging. Minimal level has to guarantee audit trails. • 100726 • Grid services must enable the measurement of quantities relevant to the verification of service level agreements (SLA)

6. Service interactions types Two types of interactions: • Service interacts with system • Configuration • Logging • Manager/Information system interacts with • service • getting service information (version number, uptime, …) • operations on service (start, stop, clear, …) • event notification (errors, full disks, ….) • container • system

7. Service interaction with System • Configuration • reading/writing of configuration • Logging • common logging format • logging to different destinations • Saving of system state • recovery after shutdown

8. Service Information • State • deployed, started, stopped • Status: running/not running • Relationships • Dependencies (e.g. needs service A, B, C) • Replacement services (is duplicate of service D,E) • Generic information • Name • Version • Package signature • Interface version • Configuration • Values • Last updates • Timing • uptime • present time • Service performance metrics • Last processing time • Longest processing time • Logging • Log information • Number of errors • Service specific internal information • Number of used DB connections • Number of requests • Available services in container / node

9. Service operation • Status operation • start, stop, pause • Internal operations • pinging • Different kind of service specific operations • cleanup connections • reset • redirect log information • add user • ….

10. Service events • Service wants to inform about special events • Service started, stopped, …. • Fatal error occurred • Internal problems • disk running full • performance low • need more coffee ….

11. The challenge … • Heterogeneous system • Web services • C++ • Java • Daemons • Common service management interfaces • basic common set for all services • can be extended by an individual service • Reuse existing capabilities – minimize impact on services • e.g. start/stop • Several information sources • service • container • node • Standard outside communication • standard outside SOAP interface: CIM, WBEM, WSDM, …. • self descriptive • Separate management from business logic • service users are interested in service API • management application are interested in management API • separate WSDL interfaces

12. Implementation scenario • We foresee a two-fold structure • one global management service per node • gateway to the outside management • collects information from different sources • Service(s) • Container(s) • System • directs commands to different destinations • Service(s) • Container(s) • System • service specific parts in each service • Provides common functionality • Loading of configuration • Interface to management service

13. Implementation scenario gLite management application Information system WBEM/CIM/WSDM via SOAP Node ServiceA Tomcat gLiteServiceManager ServiceB ? SOAP ? gLiteService JMX gLiteService service logic SOAP service logic SOAP container logic configuration log files system logic

14. Implementation scenario • Implementation scenarios • Central gLite Service Manager • Java web service • Implemented in Tomcat • JMX for internal management • Combining different sources/destinations • Outside connectors: WBEM / CIM / WSDM via SOAP … • gLite service • Java web service • inside tomcat • Use JMX for communication central gLite Service Manager • C++ web service • (extended) ServiceConfigurator for communication with central gLite Service Manager, e.g. via SOAP • Daemons • Remote/Local calls for communication with central gLite Service Manager

15. gLiteService - configuration • Example for gLiteService • Managing configuration • Load from central place • Handling of complex data type • Enable querying of configuration from outside • Example service: Datamanagement Fireman MySQL catalog (org.glite.data.catalog-service-fr-mysql) • Java webservice inside tomcat • Uses JDBC pool connection to MySQL • Sets up the JDBC connection via JNDI • Uses two classes • Service class org.glite.data.catalog.service.composite.FiremanCatalogSoapBindingImpl • DB connection class org.glite.data.common.helpers.DBManager

16. Implementation scenario Web browser HTTP HTTP adapter Tomcat gLite Manager Fireman Service gLiteService JMX JMX server ConfigurationMBean service logic SOAP configuration log files

17. Example configuration file <config> <parameters> <test.value description=“This is a test value.” value=“HelloWorld” type=“string”/> … </parameters> <container name=“egee_db” description=“EGEE Metadata Database JNDI DataSource” type=“org.apache.commons.dbcp.BasicDataSource” jndi.reference.name=“jdbc”> <parameters> <username description=“The connectionusername.” jmx.readable=“true” jmx.writeable=“false” type=“string” value=“egee”/> <driverClassName description=“The fully qualified Java class name of the JDBC driver.” jmx.readable=“true” jmx.writeable=“false” type=“string” value=“org.gjt.mm.mysql.Driver”/> … </parameters> </container> </config>

18. Turning on the central configuration Step 1: Turn on the central configuration Implement the GliteService class specifying your service name to • loads configuration from central location • here: /opt/glite/etc/config/glite-SERVICENAME.cfg.xml • makes configuration information available to • central JMX server • Data Management service

19. Turning on the central configuration ... public class FiremanCatalogSoapBindingImpl extends ServiceImpl implements FiremanCatalog { private DBManager firemanDBmanager = null; ... public FiremanCatalogSoapBindingImpl(){ super(); try{ firemanDBmanager = new DBManager(“jdbc/egee_db”); } } catch (DBException e){ ... } } ... }

20. Turning on the central configuration import org.glite.management.service.GliteService; ... public class FiremanCatalogSoapBindingImpl extends ServiceImpl implements FiremanCatalog { private DBManager firemanDBmanager = null; private GliteService gLiteService; private String serviceName = “catalog-service-fr-mysql”; ... public FiremanCatalogSoapBindingImpl(){ super(); gLiteService = GliteService.instance(serviceName); try{ firemanDBmanager = new DBManager(“jdbc/egee_db”); // firemanDBmanager = new DBManager(“egee_db”, serviceName); // firemanDBmanager = new DBManager(“egee_db”, gLiteService); } } catch (DBException e){ ... } } ... }

21. Getting the configuration Step 2: Getting the configuration information There are three possibilities • via JNDI • via JMX • via GliteService

22. Getting the configuration: via JNDI public class DBManager { protected final String m_db_pool_name; protected final DataSource m_dataSource; public DBManager(String pool) throws DBException { try { m_db_pool_name = pool; Context initCtx = new InitialContext(); Context envCtx = (Context) initCtx.lookup("java:comp/env"); m_dataSource = (DataSource) envCtx.lookup(m_db_pool_name); } catch (NamingException e) { ... } }

23. Getting the configuration: via JNDI public class DBManager { protected final String m_db_pool_name; protected final DataSource m_dataSource; public DBManager(String pool) throws DBException { try { m_db_pool_name = pool; Context initCtx = new InitialContext(); // Context envCtx = (Context) initCtx.lookup("java:comp/env"); Context envCtx = (Context) initCtx.lookup("glite"); m_dataSource = (DataSource) envCtx.lookup(m_db_pool_name); } catch (NamingException e) { ... } }

24. Getting the configuration: via JNDI • Necessary adoptions: • Change of JNDI path • Advantages • Only need to change JNDI path • Easy to change in between gLite & non-gLite system • Can use JNDI features: notification, … • Dynamic configuration possible if supported by application • Independent of central JMX server • Disadvantages • Duplication of efforts • JMX for central & remote management • JNDI for application

25. Getting the configuration: via GliteService public class DBManager { protected final String m_db_pool_name; protected final DataSource m_dataSource; public DBManager(String pool) throws DBException { try { m_db_pool_name = pool; Context initCtx = new InitialContext(); Context envCtx = (Context) initCtx.lookup("java:comp/env"); m_dataSource = (DataSource) envCtx.lookup(m_db_pool_name); } catch (Exception e) { ... } }

26. Getting the configuration: via GliteService import org.glite.management.service.GliteService; public class DBManager { protected final String m_db_pool_name; protected final DataSource m_dataSource; //public DBManager(String pool) throws DBException { public DBManager(String pool, GliteService gliteService) throws DBException { try { m_db_pool_name = pool; // Context initCtx = new InitialContext(); // Context envCtx = (Context) initCtx.lookup("java:comp/env"); // m_dataSource = (DataSource) envCtx.lookup(m_db_pool_name); m_dataSource = (DataSource) gliteService.getConfigurationParameter(m_db_pool_name); } catch (Exception e) { ... } }

27. Getting the configuration: via GliteService • Necessary adoptions: • Change JNDI lookup by GliteService function call • Pass GliteService to classes that use configuration • Advantages • Easy retrieval of values • Dynamic configuration possible if supported by application • Independent of central JMX server • Disadvantages • Extra features (e.g. notification) need to be implemented by hand • Duplication of efforts (JMX for central, JNDI for application)

28. Getting the configuration: via JMX import javax.management.*; public class DBManager { private MBeanServer mbs; public DBManager(String pool, String serviceName) throws DBException { m_db_pool_name = pool; try{ List srvList = MBeanServerFactory.findMBeanServer(null); ListIterator srvListItr = srvList.listIterator(); while(srvListItr.hasNext()) { MBeanServer localMbs = (MBeanServer) srvListItr.next(); if (localMbs.getDefaultDomain().compareTo("glite") == 0) mbs = localMbs; } ObjectName objectName = new ObjectName(serviceName + configurationSubMBeanPath); m_dataSource = (DataSource) mbs.getAttribute(objectName, m_db_pool_name); } catch (Exception e){

29. Getting the configuration: via JMX • Changes: • Lookup of central MBean server • Lookup of MBean instead of JNDI Object • Advantages • Fully integrated solution • central management and local configuration coupled • Standard management solution • Dynamic configuration possible • Can use all JMX features (notification, old/new value ..) • Disadvantages • Depends on central JMX server

30. Discussions points • Feedback • Comments • Ideas • Requirements from your side • What do you want to see in the management interface? • What requirements do you have? • Priorities? What do you want to see first? • Discussion on implementation possibilities • Architecture • Implement GliteService as • Parent class • Add-on class • JNDI / GliteService / JMX • C++ configurator • News • Feedback (Eli, …) • JMX studies • News • Feedback (Paolo, …)