1 / 24

National Datawarehouse for Traffic Information – Big Data supplier

National Datawarehouse for Traffic Information – Big Data supplier. Els Rijnierse. Contents. Introducing NDW Experiences with our big data Challenges, choices and changes. Posting.

conley
Télécharger la présentation

National Datawarehouse for Traffic Information – Big Data supplier

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. National Datawarehouse for Traffic Information – Big Data supplier Els Rijnierse

  2. Contents Introducing NDW Experiences with our big data Challenges, choices and changes

  3. Posting The last slide will ask you to post your impression, to share what struck you most with all conference attendees

  4. NDW is a Collaborative venture 24 Road authorities National 6 out of 12 provinces Cities, either independent or in an alliance Covering >6000 km road network(total Dutch road network is 130.000 km) Introducing NDW

  5. What is our aim? Develop and maintain a joint database for traffic data. Up-to-date, complete and unambiguous with known quality Create efficiency by working together and sharing information Stimulate effective use of this data for:- real time traffic management - real time traffic information - analyses, policy making and research Introducing NDW

  6. Trafficmanagement Central source for all road authorities Introducing NDW

  7. Objectives Less traffic jams Predictability Safer roads Less emission More collaboration Data voor doorstroming

  8. Happy road users Introducing NDW

  9. Supervisory Board Accountability Supervision Selection from data Common data need Supply NDW Demand Participatinggoverments (IDP) Roadauthorities Individual data supply Individual Data need Commercial parties(EDP) Service providers Infrastructure supply System provider (external) Introducing NDW

  10. Data types - 1 Traffic flow per lane per vehicle class on 14818 measuring sites Travel time (realised or estimated) per lane on 9424 measuring sites Traffic speed per lane per vehicle class on 13410 measuring sites (measuring sites may produce more kind of data) Every minute, traffic data from more than 24,000 measuring sites is collected, processed and within 75 seconds distributed to the users Introducing NDW

  11. Data collection Introducing NDW

  12. Some figures on figures Over 24,000 measurement sites Giving aprox. 460,000 figures on speed, flow and travel time each minute => >27 Million per hour => >600 million per day => >240 billion per year + meta data on these figures

  13. Real-time traffic data (February 2012: 5 cm snow) Introducing NDW

  14. Data types - 2 Road works, planned and actual Reports of congestion and accidents Status (open/closed) of bridges Near future: Status (open/closed) of peak lanes and regular lanes On occurrence data on availability of the road is collected Introducing NDW

  15. Cooperation between CBS en NDW NDW collects and distributes raw data, we do not aim to do any statistical analysis. CBS started with small NDW datasets (1 day) and is now working on a larger set (3 months) to determine new methodology Conclusion: Forget everything you learned about statistics Experiences

  16. When to start calculating (Experiences with big data – 1) When using big data: This traditional way of working does not produce statistics quicker. This requests huge datastores for raw data storage Strongly advised is starting with statistical analyses the moment data is streaming in and storing only aggregated in between results Adapt you algorithms to be able to handle correct any unpredictable gaps in the raw data that will occur Traditional statistical methodology: gather and store everything and perform the statistical analyses on certain times. Experiences

  17. Technical issues (Experiences with big data - 2) Traditional relational databases but also statistical tools (SPSS/SAS/R) are not fast enough, run far out of memory and do not have enough performance for quick retrieval of raw data. When using a data storage technique suitable for fast recovery of raw data then some coding and programming has to be done on the raw data. Recalculating because of wrong choices or methods takes an increasing amount of time as the amount of raw data grows quickly every day. Experiences

  18. Challenges, Choices, Changes Devils Triangle Contents awareness Statistical knowledge IT knowledge Challenges, Choices, Changes

  19. Challenges, Choices, Changes Challenge Government policy is that public data are open data, which means our raw data are on the WWW (www.ndw.nu/datalevering) Anybody can download them and produce surveys, statistics, tables, draw conclusions and publish these (long) before statistical office does. Be aware of publicity this might cause, discussions on ‘the truth’ and the status of a response or statement from the statistical office. Take on the challenge of producing real time statistics Challenges, Choices, Changes

  20. Challenges, Choices, Changes Choice Traditional storage of raw data used for statistics is at thestatistical office. Big data should be left at their origin and withdrawn when needed. Challenges, Choices, Changes

  21. Challenges, Choices, Changes Change Look for appropriate IT infrastructure and develop a new way of handling data Challenges, Choices, Changes

  22. www.sendsteps.com Prepare to react; keep your phone ready! Internet Go to sendc.com 1 Log in with Session 2 Type WS3 <space> your answer 3 TXT Text to +316 4250 0030 1 Type Session <space> WS3 <space> your answer 2 Posting messages is anonymous No additional charge per message

  23. When using Big Data for our statistics the biggest change in our way of working will be….

  24. Challenges, Changes and Choices when using these amounts for Statistics Forget everything you learned on statistics: How to produce 1 representative figure on traffic intensity from this:

More Related