Download
network monitoring data storage and processing n.
Skip this Video
Loading SlideShow in 5 Seconds..
Storage Architectures PowerPoint Presentation
Download Presentation
Storage Architectures

Storage Architectures

159 Views Download Presentation
Download Presentation

Storage Architectures

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Network Monitoring Data Storage and Processing Storage Architectures

  2. Introduction • Given a starting pool of data objects (e.g. raw packet files, database table of meta-data) various process can be applied to create new derived objects (either at capture time, as a batch process, or on demand). A hierarchy of processes can be applied as an automated workflow when an object (e.g. graph of throughput) is requested.

  3. Outline • Conceptual Model • Data Types and Objects • Transformations • Dependency Hierarchies and Workflows • Example • Implementation Considerations • Flow Throughput

  4. Data Types • Basic data types are those imported into the system e.g. packet traces, capture meta-data • Derived data types are those created from basic data types e.g. flow packet trace, netflow record • Some form of object-oriented hierarchy of types e.g. flow packet sequence is a packet sequence

  5. Transformations • Objects from the data store can be used to create new objects • Multiple objects can be used to create a new object • Multiple objects can be created by one transformation • Transformations can use objects inheriting the required data type e.g. flow packet sequence when packet sequence needed

  6. Transformation Parameters • Parameters for transformations are also objects • These parameter objects can be used for indexing, searching or as meta-data for the created object

  7. Dependence Hierarchies • Can request an object and if it is not in the data store then it can be created through a chain of transformations of data objects • The intermediate objects created will then be in the store, available for repeated queries or similar or overlapping queries

  8. Implementation Considerations • Performance • the transformation functions will have to be efficiently written as many TB of data are being dealt with • pre-emptive generation of objects when the object is used for other transformation • creating of common data in advance • Garbage collection due to storage limitations

  9. Example Implementation • Flow throughput line graph • Netflow records are created from pcap files for the desired time window • User selects desired flows and bin size • Throughput calculated from pcap files • Graph created from throughput data • Implemented as a Perl/CGI web app • Results cached for multiple uses

  10. Related Work • Bashir, O., Phillips, I.W., Parish, D.J., Adams, J.L. & Spencer, T. (1998), The Management and Processing of Network Performance Information, BT Technology Journal , 16(4) , October 1998, pp 203-212, ISSN: 1358-3948. • Mike Fisk, George Varghese (2002), Agile and Scalable Analysis of Network Events, IMW 2002, pp 285-290. • Root: An Object-Oriented Data Analysis Framework, CERN, http://root.cern.ch/