1 / 36

Network Measurement Virtual Observatory

István Csabai Eötvös University Budapest. Network Measurement Virtual Observatory. Science : early times. observation. theory. reality. Science : past. instruments. observation. theory. reality. experiment. models. test. predictions. Science : present. instruments. observation.

astrid
Télécharger la présentation

Network Measurement Virtual Observatory

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. IstvánCsabaiEötvös University Budapest Network Measurement Virtual Observatory

  2. Science: early times observation theory reality

  3. Science: past instruments observation theory reality experiment models test predictions

  4. Science: present instruments observation theory reality experiment models test virtual reality predictions

  5. Other disciplines are similar: whole genomes, satellite maps, sensor networks, etc. instruments observation theory reality experiment models test virtual reality predictions

  6. To understand the complex reality, we need complex models To verify complex models we need a lot of data (large statistical sample)

  7. Virtual Observatories: few key elements • Modern web and database technologies to standardize and share data • Advanced indexing techniques for multi-terabyte, multidimensional archives • Distributed computing for analysis and modeling – move computation to data • Visualization • Same challenges for different disciplines: we can share the tools and solutions

  8. Internet • Although man-made but there is no “blueprint” • “Astronomical” number of components, complex non-linear interactions • We need similar methods as in natural sciences to understand its behavior • We need to • observe / experiment • model • Future internet: self-aware, self-managing, self-healing …

  9. Measurement/experiment facilities • Planetlab / Onelab • A network of open computers distributed across the world and available for the development of new network services. • Realistic platform available for trial deployment and experimentation, with services such as distributed storage, network mapping, peer-to-peer systems, distributed hash tables and query processing. • OneLab2 EU Integrated Project 2008-2010 ,29 participants Europe wide + 1 from Japan, budget 7.5 M€

  10. Measurement/experiment facilities • Etomic • An open measurement infrastructure in Europe to carry out high temporal resolution (~10 nano second), globally synchronized, active measurements. Provides high resolution, spatially extended dynamic picture of fast changes in the network traffic.

  11. ETOMIC hardware • Server PC architecture,Linux • Endace DAG 3.6 GE card orARGOS FPGA measurement card • with packet sending capability (packetoffset ~60 ns) • GPS antenna for global time synchronization • New low cost version • based on Blackfin microcontroller • dedicated IP packet time stamping module (< μsec)‏ • Can be called as SOAP web services / from stored procedures • low cost (300 €)

  12. Web interface: experiment bundle

  13. Scheduling Time slot reservation in a calendar system for the experiments Experiments

  14. Userexperiments

  15. Science archive – based on CasJobs CasJobs: database + batch + web services (Developed for SDSS; Alex Szalay , William O’Mullane, Nolan Li, María Nieto-Santisteban, AniThakar, Jim Gray)

  16. Network Measurement Virtual Observatory • most network measurement projects: • use a single dedicated infrastructure • scan only narrow sub-segments • analyze a limited set of networkcharacteristics • centralized and separated fromeach other • key idea: try to interconnect separate • measurement efforts • large-scale behavior • long-term evolution

  17. Network Measurement Virtual Observatory • The challenge: seamless integration of multiple tools and data sources • We need easy to use standards for data access/exchange • Searchable metadata, organized access • Working prototypes, use cases • Unified interface: • standardized data model (NetXML) • wrapper software @ each platform (to convert data into NetXML) • standardized way of reaching services (Web services) • Mediator: • metadata registry • service registry (e.g. time slice service) • joint queries: query broker Many ideas, concepts from astronomical VO

  18. traceroutearchive database Mediator/querybroker GUI graphicaluser interface SQL data subset WS Processing functions Stored procedures Search tools Visualization tools Web service interface XML, SOAP WSDL, processed data delayarchive BGP archive database database graphical user interface graphicaluser interface SQL SQL datasubset data subset Processingfunctions Storedprocedures Searchtools Visualizationtools Processing functions Stored procedures Search tools Visualization tools Web service interface Web service interface XML, SOAP XML, SOAP WSDL, processed data WSDL, processed data

  19. Usecases: what do we measure • onewaydelay (60nsresolution) • tracking topology changes • available bandwidth meter • transport protocol testing • queuing delay tomography • geolocalization • …

  20. Geolocalization A ? B

  21. Geolocalization • Useful for: content distribution, user identification, network diagnostics, language selection,… • WhoIS and DNS databases are not reliable • Method: probabilistic triangulation from landmark nodes • More landmarks – more precise localization – more computation

  22. Hierarchical Triangular Mesh Hierarchical subdivision of spherical triangles, represented as quad tree. Developed for indexing the sky (SDSS; A. Szalay, G. Fekete) Standalone library + SQL Server integration Quick routines for intersecting spherical polygons: speedup geolocalization calculations

  23. Conclusion instruments observation theory reality experiment models test virtual reality predictions

  24. Conclusion instruments observation theory reality experiment models test virtual reality predictions

  25. Conclusion instruments observation theory reality experiment models test virtual reality predictions

  26. Collaborators, acknowledgements • P. Mátray, P. Hága, S. Laki, J. Stéger B. Hullár, G. Vattay (Eötvös University) • collaborations • Universidad Autonoma de Madrid • Universidad Publica de Navarra • Ericsson Research • Tel Aviv University This work was partially supported by the National Office for Research and Technology (NAP 2005/ KCKHA005) and the EU ICT Moment (#215225) and OneLab2 Integrated Project(#224263).

  27. Sokboldogszületésnapot, Sanyi!

  28. Plans • Probabilistic geolocation • Using the distribution of the velocity of signal propagation • Pre-clustering delay measurements Sándor Laki (C) IP Geolocation

  29. New kind of science:mega-surveys • New technology has allowed new kind of surveys • All-sky surveys • GALEX ,SDSS, IRAS,WMAP, NVSS, … • Multi-wavelength • Radio, infrared, optical, UV, X-ray, Gamma-ray • Gravitational waves, neutrinos & cosmic particles • Time domain • Pan-STARRS, LSST, GAIA, … • Collecting data in (public) digital archives • Instead of targeting special objects: observe everything • New challenge: store, organize, analyze huge amount of data • Virtual Observatories / E-science

  30. main goals: • create large archives to: • share available measurement data & analysis results • provide easy-to-use “online” network data analysistools • integrate existing measurement and monitoring infrastructures toward a common and open platform: • standardisation of network measurement data • design and implement a system to share network data

  31. store & share raw data • joint analysis of different types of measurement data • reanalysis (with new statistical methods) • reference data (historical comparison) • share analysis tools • server side processing simplifies client applicatons • no need to transfer bulk data packages: online processing • exchange data in a standardized way • existing efforts (OGF, IETF, KML) • Network Measurement Virtual Observatory (nmVO) Database Web services netXML

  32. Science archive – based on CasJobs database Stored procedure Web service interface

  33. Open repository

More Related