1 / 31

Navigating Big Data: The Cognitive Bottleneck in Understanding and Interpretation

The intersection of big data and human cognition presents unique challenges for data scientists. While big data is abundant, the true bottleneck lies not in computational power but in our ability to interpret and understand the insights derived from this data. This discussion covers the importance of a people-centric approach to data interpretation, the trade-offs between accuracy and interpretability, and the impact of speed and interaction on user experience. Join the conversation on enhancing the interplay between big data and human intellect for deeper insights.

abigail-puckett
Télécharger la présentation

Navigating Big Data: The Cognitive Bottleneck in Understanding and Interpretation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. BIG DATA,We have a communication problem. GINORMOUS SYSTEMS April 30–May 1, 2013 Washington, D.C. Daniel Tunkelang Head of Query Understanding, LinkedIn

  2. BIG DATA IS EVERYWHERE

  3. BIG DATA POWERS EVERYTHING

  4. DATA SCIENTISTS WORRY ABOUT VOLUME, VELOCITY, VARIETY, …

  5. BUT THE BOTTLENECK ISN’TCOMPUTATIONAL IT’S COGNITIVE

  6. BIG DATA IS A TOOL Doug Engelbart, inventor of the mouse, hypertext, etc. TOOLS AUGMENT HUMAN INTELLECT

  7. NOT EVERYONE SUBSCRIBES TO THIS POINT OF VIEW… Claudia Perlich, Chief Scientist of media6degrees, speaking at TTI/Vanguard 2012 Conference on Understanding Understanding:

  8. SHE HAS A POINT

  9. BUT PREDICTIVE MODELING ISNOT ENOUGH

  10. TRAINING DATA? OBJECTIVE FUNCTION?

  11. WE NEED A PEOPLE-CENTRIC APPROACH TO BIG DATA INTERPRETABILITY INTERACTION INSIGHT

  12. LET’S START WITH INTERPRETABILITY

  13. EXAMPLE: SVM vs. DECISION TREE

  14. DECISION TREES HAVE FLAWS… DISCRETE

  15. early splits provide big picture… …or reveal training data problems BUT THEY COMMUNICATE (if they’re shallow) fat leaves guide feature engineering

  16. WHICH SUPPORTS ITERATION

  17. INTERPRETABILITY DELIVERS • Key search leader favors rule-based approach for key scoring algorithms. • Replaced regression with decision tree in local search model: gained accuracy and insight. • Using trees to recognize spam, analyze search abandonment, model / quantify social proof.

  18. GO DEEP vs INTERPRETABILITY A KEY DATA SCIENCE TRADE-OFF

  19. ON TO INTERACTION

  20. DON’T OVERPAY FOR PRECISION

  21. BE FAST, CHEAP, AND 98% RIGHT http://metamarkets.com/2012/fast-cheap-and-98-right-cardinality-estimation-for-big-data/

  22. ARE PEOPLE THAT IMPATIENT? tolerable wait time for web users 0.1s increase in latency significantly reduces # of searches, ad revenue tl;dr: YES

  23. IMPATIENCE IS GOOD SPEED MATTERS

  24. INSIGHT

  25. http://blog.takejune.com/archives/52334044.html

  26. BE TRENDY AND NORMALIZE vs

  27. SOLVE FOR INTERESTINGNESS Sept. 11th Abu Ghraib Weapons Inspectors

  28. COMPUTE POTENTIAL INSIGHTS APPLY HUMAN INTUITION

  29. SUMMARY: Let’s have a conversation with Big Data. INTERPRETABILITY INTERACTION INSIGHT

More Related