1. LHCb Logging System A computer log is a diary or archive of events, in this case generated by a computer system or systems. In the late- 23rd century, Federation starships were equipped with a "black box" that stored computer logs. The logs could be used in criminal investigations or to determine the cause of a lost ship. Computer logs were for official purposes only and were available to authorities only under specific legal circumstances or court-order. Nikolaidis Fotis ( fotis.nikolaidis @ cern.ch ) University Of Crete, Greece

2. Sources Web Servers Gateways Network Components Farm Nodes PVSS FMC

3. Storage Schema hlt[a-e][1-11] Messages, crond, maild, dnsd, secure, secureNagios FARM HOSTS hostName Messages, crond, maild, dnsd, secure, secureNagios {Other files either from FMC or web sites} PVSS_II.log PVSS00ctrl50.log And other ... PVSS hostName Project Name LHCb TFC FEST ECAL ........ DAQ $partition.log PARTITIONS Messages, crond, maild, dnsd, secure, secureNagios TELL1 SERVICES Dataremove, Dimrpc, Writerd, Xmlrpc

4. Needs Forensic / Troubleshooting Splunk ( http://admin01/splunk ) Real Time Alert Ossec

5. Splunk Is a High performance, scalable software server written in C/C++ and Python. Index and Normalize logs (disk fail , disk error are the same) Can be combined with with Ossec, Snort and other IDS via plugins Does not need an external Database.

6. Splunk - Features Advanced search Regular Expressions / Time Windows Runtime statistical analysis Extensible Modules, Patterns Dashboards

7. Splunk - More Features Can correlate events of different hosts/formats Supports many log formats out of the box (For non standard logs such as FMC configuration is needed) If run on CLI , can be integrated to scripts

11. Have a closer look here ...

12. The first line is excluded The second line is now the first

13. Who is keeping ssh busy ? ;p

14. New Patterns can be generated almost automatically

15. Internal Information

16. OSSEC

17. OSSEC Open Source Host-based Intrusion Detection System. Log analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting and active response.

18. OSSEC Analyzes incoming logs runtime and reacts if needed Every event can be ranked with a value [1-14] If event > mailRank , send a mail If event > scriptRank , execute a script Rules are defined in XML files Message, frequency, priority, etc

19. Fault Tolerance

20. Normal

21. Logsrv01 failure

22. Log analysis failure

23. Logsrv02 failure