(Ab-) using Round Robin Databases with Perl

1. (Ab-) using Round Robin Databases with Perl Mike Schilli, Yahoo! 06/18/2008

2. Overview • What are Round Robin Databases? • RRD and Perl – which module to use? • Practical Examples

3. Round Robin Databases • Archives of fixed size for unlimited data • Overwrite old spots if full

4. Round Robin Archives start

5. Round Robin Archives • Overwrite if full

6. Round Robin Archives • RRDs with multiple archives with different granularity

7. Data Flow with rrdtool Data Source Archive

8. Data Source Conversions

9. Steps and Heartbeat

10. Steps and Heartbeat

11. Steps and Heartbeat

12. Generous Heartbeat

13. Generous Heartbeat

14. Steps and Heartbeat • Long story short: Heartbeat > Step if you’re roughly feeding in “step” intervals. • Example: • Step=300 (5 min) • Heartbeat=600

15. Feeding PDPs to the Archive

16. Feeding PDPs to the Archive • Consolidation Functions (MAX, AVERAGE, etc.) • xfiles factor (allowed percentage of missing values)

18. Simple Example Web Orders: 0:00 100 1:00 110 2:00 120 3:00 130 4:00 140 5:00 150

19. Simple Example

20. Simple Example rrdtool 'create' 'myrrdfile.rrd' '--start' '1211871600' '--step' '3600' 'DS:orders:GAUGE:7200:U:U' 'RRA:MAX:0.5:1:100' rrdtool 'update' 'myrrdfile.rrd' '1211875200:100' rrdtool 'update' 'myrrdfile.rrd' '1211878800:110' rrdtool 'update' 'myrrdfile.rrd' '1211882400:120' rrdtool 'update' 'myrrdfile.rrd' '1211886000:130' rrdtool 'update' 'myrrdfile.rrd' '1211889600:140' rrdtool 'update' 'myrrdfile.rrd' '1211893200:150' rrdtool 'graph' 'mygraph.png' '--vertical-label' 'Orders per Hour' '--end' '1211893200' '--start' '1211871600' 'DEF:draw1=myrrdfile.rrd:orders:MAX' 'AREA:draw1#0000FF:Web Orders

21. RRDTool • Arcane syntax • Perl API RRDs.pm • Pretty close to RRDTool syntax • Calls rrdtool under the hood

22. To the Rescue: RRDTool::OO • CPAN Module RRDTool::OO • Takes the drudgery out of the RRD syntax • Does what you want by default • Translates between Perl and RRD (watch it with debug on) • Named parameters instead of fixed order

23. Simple Example #!/usr/local/bin/perl -w use strict; use RRDTool::OO; use DateTime; my $rrd = RRDTool::OO->new( file => "myrrdfile.rrd" );

24. Simple Example my $start_dt = DateTime->today( time_zone => "local" ); my $dt = $start_dt->clone();

25. Simple Example $rrd->create( step => 3600, start => $start_dt->epoch(), data_source => { name => "orders", type => "GAUGE", }, archive => { rows => 100 } );

26. Simple Example for my $hour (0..5) { $dt->add(hours => 1); $rrd->update( time => $dt->epoch(), value => 100 + $hour*10, ); }

27. Simple Example $rrd->graph( image => "mygraph.png", vertical_label => 'Orders per Hour', start => $start_dt->epoch(), end => $dt->epoch(), draw => { type => "area", color => '0000FF', legend => "Web Orders", } );

28. Simple Example

29. RRDTool Commands rrdtool 'create' 'myrrdfile.rrd' '--start' '1211871600' '--step' '3600' 'DS:orders:GAUGE:7200:U:U' 'RRA:MAX:0.5:1:100' rrdtool 'update' 'myrrdfile.rrd' '1211875200:100' rrdtool 'update' 'myrrdfile.rrd' '1211878800:110' rrdtool 'update' 'myrrdfile.rrd' '1211882400:120' rrdtool 'update' 'myrrdfile.rrd' '1211886000:130' rrdtool 'update' 'myrrdfile.rrd' '1211889600:140' rrdtool 'update' 'myrrdfile.rrd' '1211893200:150' rrdtool 'graph' 'mygraph.png' '--vertical-label' 'Orders per Hour' '--end' '1211893200' '--start' '1211871600' 'DEF:draw1=myrrdfile.rrd:orders:MAX' 'AREA:draw1#0000FF:Web Orders

30. Where RRDtool shines • Graph time-based data • Don’t worry about scaling or axis text • Stacked graphs

31. Where RRDtool sucks • Can’t add data for past events • Can’t add data twice at the same timestamp

32. Pitfalls • Make sure that the first update happens after RRD “start” time • Updates need to happen at increasing timestamps • On large step sizes (1 year), limit the number of rows to fit within < 2030.

33. Debugging with Log4perl use Log::Log4perl qw(:easy) Log::Log4perl-> easy_init( $DEBUG );

34. Log4perl Output rrdtool 'create' 'myrrdfile.rrd' '--start' '1211871600' '--step' '3600' 'DS:orders:GAUGE:7200:U:U' 'RRA:MAX:0.5:1:100' rrdtool 'update' 'myrrdfile.rrd' '1211875200:100' rrdtool 'update' 'myrrdfile.rrd' '1211878800:110' rrdtool 'update' 'myrrdfile.rrd' '1211882400:120' rrdtool 'update' 'myrrdfile.rrd' '1211886000:130' rrdtool 'update' 'myrrdfile.rrd' '1211889600:140' rrdtool 'update' 'myrrdfile.rrd' '1211893200:150' rrdtool 'graph' 'mygraph.png' '--vertical-label' 'Orders per Hour' '--end' '1211893200' '--start' '1211871600' 'DEF:draw1=myrrdfile.rrd:orders:MAX' 'AREA:draw1#0000FF:Web Orders

35. Example: Stock Portfolio

36. Example: Measure Power Usage • Mastech MAS-345 • Serial Interface (RS-232)

37. Example: Measure Power Usage

38. Example: Measure Power Usage

39. Aberrant Behaviour

40. The End Thanks!

41. References • RRDTool::OO on CPAN • RRD::Simple on CPAN • Portfolio Rendering article: http://perlmeister.com/talks/lm-200805.pdf • RRDTool article: http://www.linux-magazine.com/issue/44/Perl_RDDtool.pdf • RRDTool article (in German): http://www.linux-magazin.de/Artikel/ausgabe/2004/06/perl/perl.html

42. Bonus Slides

43. Aberrant Behaviour • Holt Winter Algorithm • Data Smoothing/Prediction

44. Aberrant Behaviour • Holt Winter Algorithm • RRAs: • HWPREDICT • SEASONAL • DEVSEASONAL • DEVPREDICT • FAILURES • RRA:HWPREDICT:rows:alpha:beta:seasonal_period

45. Aberrant Behaviour • alpha (baseline): 0..1 • 1: recent observations carry greater weight • 0: past observations carry greater weight • beta (slope): 0..1 • gamma (seasonal): 0..1 • failures defaults to 7

46. Aberrant Behaviour • Example: • alpha = 0.50 • beta = 0.50 • gamma = 0.01 • seasonal_period = 2 • threshold = 7 • window_length = 9