Network Monitoring Tools Overview: RGMA Producers and Archiver

R-GMA Producers and Archiver Paul Mealor

gsiwuftp Overview edg-netmon-archiver edg-gridftplog2rgma Log Archiver API P Producer API MySQL DB Here be servlets edg-netmon-rgma-info P Producer API P A C DBP edg-pinger-timeping edg-netmon2rgma Producer API FIFO

edg-netmon2rgma • Maintains producers for the tools • Tools’ scripts are rerun for each measurement batch • Program running APIs must be persistent • Files • /opt/edg/sbin/edg-netmon2rgma.pl • /opt/edg/var/edg-netmon2rgma/edg-netmon2rgma-fifo • /etc/rc.d/init.d/edg-netmon2rgmad • Configuration • RGMA client configuration

Tool/netmon2rgma interaction PingER Normal log Web interface • addmetric pinger rtt 10 NMIdSource=pc18.hep.ucl.ac.uk tool=pinger edg-netmon2rgma If producer doesn’t exists If bufferSize < count * 4 / 3 • CircularBufferProducer_new(“NetworkRTT”, • “WHERE NMIdSource=‘pc18.hep.ucl.ac.uk’ AND tool=‘pinger’”); • CircularBufferProducer_setRemoteBufferSize($producer,10*2);

Tool/netmon2rgma interaction PingER Normal log Web interface • measurement pinger rtt pc18.hep.ucl.ac.uk edgnm.gridpp.rl.ac.uk pinger 100 57 edg-netmon2rgma • CircularBufferProducer_insert($producer, • ”INSERT INTO NetworkRTT • (NMIdSource, NMIdDestination, tool, packetSize, value) • VALUES • (pc18.hep.ucl.ac.uk, edgnm.gridpp.rl.ac.uk, pinger, 100, 57)”);

edg-netmon2rgma • Possibly should rewrite in Java • Problem in C (Perl) API results in memory leak and heavy performance hit • Current solution is to restart the daemon with a cron job • /etc/cron.d/edg-netmon2rgma-restart • But java lacks the text-handling of Perl • Would have to have java on the NM

Tools modifications • Generally 2 new functions • initialise_rgma • Writes initialisation directives to the FIFO • i.e. fixed columns, metric names, expected number of entries • Netmon2rgma creates producers, changes buffer sizes when necessary • produce_it • Writes the results of the measurements • No extra configuration • Assumes /opt/edg/var/edg-netmon2rgma/edg-netmon2rgma-fifo

edg-netmon-rgma-info • Publishes NetworkSE, NetworkCE tables • Republishes every 12 hours (i.e. keepalive) • Files • /opt/edg/sbin/edg-netmon-rgma-info.pl • /opt/edg/etc/edg-netmon-rgma-info.conf • Local NM, associated SEs and CEs • /etc/rc.d/init.d/edg-netmon-rgma-infod • RGMA client configuration

edg-ftlog2rgma • Publishes file-transfer logs (i.e. GridFTP) • Files • /opt/edg/sbin/edg-gridftplog2rgma.pl • /etc/rc.d/init.d/edg-gridftplog2rgmad • Reads from GridFTP log • /var/log/gsiwuftpd.log • Publishes new entries when they are added • RGMA client configuration

edg-netmon-archiver • Program • /opt/edg/share/edg-netmon-archiver.jar • /opt/edg/sbin/edg-netmon-archiver • runs the Java code • /opt/edg/bin/edg-netmon-archiver-setup • Configures MySQL database; requires mysql root password • What to do in LCFG? • /opt/edg/etc/edg-netmon-archiver.conf • List of tables to archiver plus predicates for each • /etc/rc.d/init.d/edg-netmon-archiverd • Also RGMA client configuration

edg-netmon-archiver • Ironically this might be the best package to write in C or Perl • Archiver API is all but unaffected by C API problem • But might run on R-GMA servlet machine • Interaction with edg-netmon-info • Archives every row that edg-netmon-info produces • Perhaps need to automatically remove duplicates (via mysql)

Package progress • Tools, edg-netmon2rgma, edg-netmon-rgma-info • 99% • Latest tweaks still need to be uploaded • edg-ftlog2rgma • 75% • rc script, autobuild process • edg-netmon-archiver • 95% • need to upload, tag, check build process (ant)

Testing • Tools on pc18.hep.ucl.ac.uk • edg-netmon2rgmad on pc18.hep.ucl.ac.uk • “User servlets” on pc23.hep.ucl.ac.uk • Registry and Schema servlets on pc23.hep.ucl.ac.uk • Something always goes wrong when I try to connect to the WP3 Registry at infocat.gridpp.rl.ac.uk • >1200 RTT/loss measurements archived by Tuesday • >130 TCP throughput measurements to localhost :)

GridFTP logging in MDS Paul Mealor

Status • Information provider • Will be in EDG 2 on SE • Publishes the most recent information for each host pair • MDS being phased out in EDG 2 • GIIS structure replaced by R-GMA • Will be all R-GMA later

Details • Publishes achieved throughput • Filesize, time etc as parameters • Publishes most recent transfer between each host pair • Specifies source and host pair, plus, if known, source and host NM • NM names taken from config file as with other netmon info providers • Uses same schema as other netmon • But doesn’t use the full tree structure

Schema dn: NMMeasureId=$source/$dest,$opt{rootdn} objectClass: NetworkMeasurement NMMeasureId: $source/$dest SourceHost: $source SourceNE: $sourceNE DestHost: $dest DestNE: $destNE MetricName: gridftpthroughput MetricValue: $throughput MetricUnit: bit/s MonitorTool: GridFTP MetricTime: $timestamp Parameter: time: $etime Parameter: size: $size Parameter: buffersize: $buffer Parameter: streams: $streams Parameter: stripes: $stripes createTimestamp: $timestamp modifyTimestamp: $timestamp

END

Network Monitoring Tools Overview: RGMA Producers and Archiver