What to Do? A Research Agenda

1. What to Do? A Research Agenda Jim Gray Microsoft Research

2. “Everything that can be invented has been invented.”Commissioner, U.S. Office of Patents, 1899 From http://inventors.about.com/library/lessons/bl_appendix5.htm…absolutely no basis to support Duell's alleged statement. Just the opposite is true.Duell's 1899 report documents an increase of about 3,000 patents over the previous year, and nearly 60 times the number granted in 1837. Further, Duell quotes President McKinley's annual message saying, "Our future progress and prosperity depend upon our ability to equal, if not surpass, other nations in the enlargement and advance of science, industry and commerce. To invention we must turn as one of the most powerful aids to the accomplishment of such a result." Duell adds, "May not our inventors hopefully look to the Fifty-sixth Congress for aid and effectual encouragement in improving the American patent system?" These are unlikely words of someone who thinks that everything has been invented.

3. “We have patents on the Byte and the Algorithm”Dave Huffman • EE has the electron • Physics has matter/energy • Chemistry has molecules and reactions • Economics has the transaction. • They all need us to do anything • The IT revolution is just starting.

4. 1. Introduction: Scientific Progress is Essential Science is a Proper Concern of Government Government Relations to Science - Past and Future Freedom of Inquiry Must be Preserved 2. The War Against Disease: In War In Peace Unsolved Problems Broad and Basic Studies Needed Coordinated Attack on Special Problems Action is Necessary 3. Science and the Public Welfare: Relation to National Security Science and Jobs The Importance of Basic Research Centers of Basic Research Research Within the Government Industrial Research International Exchange of Scientific Information The Special Need for Federal Support The Cost of a Program 4. Renewal of our Scientific Talent: Nature of the Problem A Note of Warning The Wartime Deficit Improve the Quality Remove the Barriers The Generation in Uniform Must Not be Lost A Program 5. A Problem of Scientific Reconversion: Effects of Mobilization of Science for War Security Restrictions Should be Lifted Promptly Need for Coordination A Board to Control Release Publication Should be Encouraged 6. The Means to the End: New Responsibilities for Government The Mechanism Five Fundamentals Military Research National Research Foundation Science The Endless FrontierVannevar Bush -> Harry Truman, July 1945 http://www.nsf.gov/od/lpa/nsf50/vbush1945.htm

5. Request From George Djorgovski • What do you think we (CACR) should do? • Long term • 2 or 3 focus areas • Leverage our skills.

6. Honest Answer • I do not know. • Hire smart over-achievers • Give them (barely) enough resources • Ask them what they have accomplished • Foster mutual respect • Take credit for the successes • Allow some of them to fail.

7. Talking the Talk and Walking the Walk • We have 7 people – what’s our agenda? • Observe it is all about:Data → Information → Knowledge → WisdomPeople == Communication is the “killer app” • ½ Personal Information Management • ½ Corporate Information Management

8. MemexAs We May Think, Vannevar Bush, 1945 “A memex is a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility” “yet if the user inserted 5000 pages of material a day it would take him hundreds of years to fill the repository, so that he can be profligate and enter material freely”

9. 25Kday life ~ Personal Petabyte 1PB Will anyone look at web pages in 2020? Probably new modalities & media will dominate then.

10. Challenges • Capture: Get the bits in • Organize: Index them • Manage: No worries about loss or space • Curate/ Annotate: automate where possible • Privacy: Keep safe from theft / disclosure. • Summarize: Give thumbnail summaries • Interface: how ask/anticipate questions • Present: show it in understandable ways.

11. Radio capture tool Telephone capture tool PocketPC transfer tool PocketRadio player TV capture tool Radio EPG tool TV EPG download tool MAPI interface Legacy email client Browser tool Internet files Legacy applications MyLifeBits Shell IM capture Voice annotation tool Text annotation tool Import files MyLifeBits Software MyLifeBits store database

12. MyLifeBits Interesting Ideashttp://www.research.microsoft.com/barc/mediapresence/MyLifeBits.aspx • Capture is “easy” • All TV, all Phone, all web pages, all mail,…. • Senscam: a picture a minute • GPS corrleation (and other correlation) • Interesting search strategies • Pivot and cluster • Struggling with metadata (annotations) • Interesting visualizations (ambiance, timelines, spatial, conceptual). • Very topical interest from public, industry, developers

13. Relevance To CACR • (re)Invent the laboratory notebook • Scientists are still mostly working with paper and pencil (especially in the lab). • Change the way scientists do information management. • Give them good analysis/visualization tools • Allow them to publish their data/workbooks

14. 80% of data is personal / individual. But, what about the other 20%? • Business • Wall Mart online: 1PB and growing…. • Paradox: most “transaction” systems < 1 PB. • Have to go to image/data monitoring for big data • Government • Government is the biggest business. • Science • LOTS of data.

15. Data Challenges I'm Struggling With • Sneakernet is probably the best way to moving WAN data at 1GBps File transfer efforts are currently 550MBps via Internet2. How to manage the multi-petybyte file repository we are about to generate. • The TerraServer has evolved from a mainframe to a bunch of bricks. The new design has been operating for a year and we are quite pleased with it. But we face "how-do-you-manage a bunch?" and what the best geoplex strategy?. • The SkyServer website is built using database technology and web services. Now moving the web services inside the database. Others are working to design a scale-out version of the server. There are several interesting data challenges in these changes. • Using relational tuples to represent spatial volumes as constraints. Point-in-polygon and polygon-overlap queries can then be quickly evaluated. I will briefly describe this idea.

16. SpeedMbps Rent$/month $/TBSent Context $/Mbps Time/TB 0.04 40 1,000 3,086 6 years Home phone 50 Home DSL 0.6 117 360 5 months T1 1.5 1,200 800 2,469 2 months T3 43 28,000 651 2,010 2 days OC3 155 49,000 316 976 14 hours OC 192 9600 1,920,000 200 617 14 minutes 100 Mpbs 100 1 day Gbps 1000 2.2 hours How Do You Move A Terabyte? Source: TeraScale Sneakernet, Microsoft Technical Report May 2002, MSR-TR-2002-54 http://research.microsoft.com/research/pubs/view.aspx?tr_id=569

17. Moving Data Bricks • WAN costs >> 100$/Mbps/month>> 1$/GB • Beowulf networking 10,000x cheaper than WAN factors of 105 matter. • The cheapest and fastest way to move a Terabyte cross country is sneakernet.24 hours = 4 MB/s50$ shipping vs 1,000$ wan cost.

18. OC192 = 9.9 Gbps Giga Byte Per Second File Mover • CERN to Pasadena • Windows TCP/IP stack improvements • Opteron demo • Disk-to-Disk at 550MBps now (~2 TB/Hour) • Near the PCI-X limit. • GOAL: 1GBps disk-to-disk • 75% there PCI -X limit tcp limit

19. But then what?Managing Petabytes • CERN files are 30MB • They produce 1 B files/year. • How name them? • How manage them? • Depends on workload: how use them. • It’s a DB problem.

20. Data Challenges I'm Struggling With • Sneakernet is probably the best way to moving WAN data at 1GBps File transfer efforts are currently 550MBps via Internet2. How to manage the multi-petybyte file repository we are about to generate. • The TerraServer has evolved from a mainframe to a bunch of bricks. The new design has been operating for a year and we are quite pleased with it. But we face "how-do-you-manage a bunch?" and what the best geoplex strategy?. • The SkyServer website is built using database technology and web services. Now moving the web services inside the database. Others are working to design a scale-out version of the server. There are several interesting data challenges in these changes. • Using relational tuples to represent spatial volumes as constraints. Point-in-polygon and polygon-overlap queries can then be quickly evaluated. I will briefly describe this idea. Jim Gray, Microsoft Research

21. TerraServer / TerraServicehttp://terraService.Net/http://TerraServer-USA.com/ • USGS Photo of US • Online since June 1998 • Operated by Microsoft • 20 TB data source • 10 M web hits/day • A web service • Our laboratory

22. KVM / IP TerraServer – What’s new • Web Service and Web Server • New ~1 ft2/pixel full color image of 120 urban areas • Storage Bricks • Commodity servers” • 4 TB raw / 2 TB Raid1 SATA storage • Dual 2 GHz + 4GB RAM • 3 Bricks = TerraServer data • Data partitioned • Moving to Yukon • Working on low TCO auto-manage • Low Cost Availability Pair & Spare • RAID1 Mirroring • Mirrored Bunches (Yukon log ship?) • Spare Brick • Web Application • Load balances mirrors • Uses surviving database on failure

23. TerraServer Challenges • Best Geoplex strategy? • Moving Web Services into the DB? • Managing bunches (lower TCO).

24. World Wide Telescope • Premise: Most Astronomy data is online • So, the Internet is the world’s best telescope: • It has data on every part of the sky • In every measured spectral band: • As deep as the best instruments (2 years ago). • It is up when you are up.The “seeing” is always great (no working at night, no clouds no moons no..). • It’s a smart telescope: links objects and data to literature on them.

25. SkyServer.SDSS.orgBuilt with Johns Hopkins U. • A modern archive • Raw Pixel data lives in file servers • Catalog data (derived objects) in Database • Online query to any and all • Also used for education • 150 hours of online Astronomy • Implicitly teaches data analysis • Interesting things • Spatial data search • Query interface via Emacs, Perl, Java… • Popular -- 1% of Terraserver  • Cloned by other surveys (a template design) • Based on Web Services

26. Quick Overview (Services) • SkyServer (skyserver.sdss.org) • Web site delivers Sloan Digital Sky Survey data • Also has education • 1,000x less popular than Terraserver, but HUGE for a science website. • A Batch Job System with Personal DBs • Lets users run jobs http://casjobs.sdss.org/CasJobs/ • Parameters & Answers to & from Personal DB • Simple batch job scheduler. • Web Services: http://www.voservices.org/ • Photographic objects • Spectrographic objects • Transformation functions • 7 out of the 8 are .NET.

27. ImageCutout SkyQuery Portal 2MASS INT SDSS FIRST Federation: SkyQuery.Net • Combines 15 archives • Send query to portal, portal joins data from archives. • Evolving Portal to have • Personal databases (workbenches) • Batch scheduling of monster queries.

28. The Data Challenges • Parallel data search (data pump).How to partition?How manage load • Moving web services to DB What is the right approach? • Move objects into DBSpatial access methodsData analysis in the DB. • Managing Petabytes

29. The Knowledge Challenge • We need to “objectify science” • What is a Gene? Star? River? Cell? … • What is the definition (formal) • What attributes do they have? • What are their dynamics? • Defining this (the “O” word) has to happen in each discipline. • This is the Knowledge level (rather than the data level)

30. Data Challenges I'm Struggling With • Sneakernet is probably the best way to moving WAN data at 1GBps File transfer efforts are currently 550MBps via Internet2. How to manage the multi-petybyte file repository we are about to generate. • The TerraServer has evolved from a mainframe to a bunch of bricks. The new design has been operating for a year and we are quite pleased with it. But we face "how-do-you-manage a bunch?" and what the best geoplex strategy?. • The SkyServer website is built using database technology and web services. Now moving the web services inside the database. Others are working to design a scale-out version of the server. There are several interesting data challenges in these changes. • Using relational tuples to represent spatial volumes as constraints. Point-in-polygon and polygon-overlap queries can then be quickly evaluated. I will briefly describe this idea.

31. A Detail: 3 Ways We Do Spatial? • Hierarchical mesh (extension to SQL) • Uses table valued stored procedures • Acts as a new “spatial access method” • Porting to Yukon CLR for a 10x speedup. • Zones: fits SQL like a glove • Amazingly simple, amazingly good. • Constraints: a really novel idea • Lets us do algebra on regions. • Paper:There Goes the Neighborhood: Relational Algebra for Spatial Data Search • Idea in backup slides.

32. y x=c/a x ax + by = c y=c/b y x Equations Define Subspaces • For (x,y) above the lineax+by > c • Reverse the space by-ax + -by > -c • Intersect a 3 volumes: a1x + b1y > c1a2x + b2y > c2a3x + b3y > c3

33. HTM Approach • Table-valued function find points near a point • Select * from fGetNearbyEq(ra,dec,r) • Use Hierarchical Triangular Mesh www.sdss.jhu.edu/htm/ • Space filling curve, bounding triangles… • Standard approach • 13 ms/call… So 70 objects/second. • Too slow, so precompute neighbors: Materialized view. • At 70 objects/sec it takes 6 months to compute a billion objects.

34. Areas defined by String(or struct in C# world) • circleSpec := CIRCLE J2000 ra dec radArcMin • | CIRCLE CARTESIAN x y z radArcMin • rectSpec := RECT J2000 {ra dec}2 • polySpec := POLY J2000 {ra dec}3+ • | POLY CARTESIAN { x y z }3+ • hullSpec := CHULL J2000 {ra dec}3+ • | CHULL CARTESIAN { x y z }3+ • convexSpec := CONVEX { x y z d}+ • regionSpec := REGION { convexSpec }+ • areaSpec := circleSpec | rectSpec | polySpec • | hullSpec | regionSpec

35. Working in 3D Avoids Spherical Geometry • P = (px,py,pz) Inside circle C centered at x,y,z,With radius r radians ifP•C > cos(r) • Arbitrary polygons are intersections of these regions. Cos(r)

36. Find Points Inside Area • fGetNearbyObjEq(ra,dec,r)fGetNearbyObjXyz(x,y,z,r)fGetNearestObjEq(ra,dec,r)fGetNearestObjEq(x,y,z,r) • fGetObjInside(region) • Recently Alex added Healpix,Igloo is also a nice iso-area decomposition

37. HTM reprise • Good for point-in-area. • Not good for area-overlaps-area(but can simplify areas and test for empty)

38. y x=c/a x ax + by = c y=c/b y x To Repeat (for area algebra) Equations Define Subspaces • For (x,y) above the lineax+by > c • Reverse the space by-ax + -by > -c • Intersect a 3 volumes: a1x + b1y > c1a2x + b2y > c2a3x + b3y > c3

39. Domain is Union of Convex Hulls Not a convex hull • Simple volumes are unions of convex hulls. • Higher order curves also work • Complex volumes have holes and their holes have holes. (that is harder). +

40. Now in Relational Terms create table HalfSpace ( domainID int not null -- domain name foreign key references Domain(domainID), convexID int not null, -- grouping a set of ½ spaces halfSpaceID int identity(), -- a particular ½ space x float not null, -- the (a,b,..) parameters y float not null, -- defining the ½ space z float not null, c float not null, -- the constant (“c” above) primary key (domainID, convexID, halfSpaceID) (x,y,z) inside a convex if it is inside all lines of the convex (x,y,z) inside a convex if it is NOT OUTSIDE ANY line of the convex select convexID -- return the convex hulls from HalfSpace -- from the constraints where @x * x + @y * y + @x * z < l -- point outside the line? group by all convexID -- consider all the lines of a convexID having count(*) = 0 -- count outside == 0

41. The Algebra is Simple (Boolean) @domainID = spDomainNew (@type varchar(16), @comment varchar(8000)) @convexID = spDomainNewConvex (@domainID int) @halfSpaceID = spDomainNewConvexConstraint (@domainID int, @convexID int, @x float, @y float, @z float, @l float) @returnCode = spDomainDrop(@domainID) select * from fDomainsContainPoint(@x float, @y float, @z float) Once constructed they can be manipulated with the Boolean operations. @domainID = spDomainOr (@domainID1 int, @domainID2 int, @type varchar(16), @comment varchar(8000)) @domainID = spDomainAnd (@domainID1 int, @domainID2 int, @type varchar(16), @comment varchar(8000)) @domainID = spDomainNot (@domainID1 int, @type varchar(16), @comment varchar(8000))

42. What! No Bounding Box? • Bounding box limits search.A subset of the convex hulls. • If query runs at 3M halfspace/sec then no need for bounding box, unless you have more than 10,000 lines. • But, if you have a lot of half-spaces then bounding box is good.

43. Zone Approach • Divide space into zones • Key points by Zone, offset(on the sphere this need wrap-around margin.) • Point search look in a few zones at a limited offset: ra ± r a bounding box that has 1-π/4 false positives • All inside the relational engine • Avoids “impedance mismatch” • Can “batch” all-all comparisons • 33x faster and parallel6 days, not 6 months! r ra-zoneMax x √(r2+(ra-zoneMax)2) cos(radians(zoneMax)) zoneMax Ra ± x

44. In SQL select o1.objID -- find objects from zone o1 -- in the zoned table where o1.zoneID between -- where zone # floor((@dec-@r)/@zoneHeight) and -- overlaps the circle floor((@dec+@r)/@zoneHeight) and o1.ra between @ra - @r and @ra + @r -- quick filter on ra and o1.dec between @dec-@r and @dec+@r -- quick filter on dec and ( (sqrt( power(o1.cx-@cx,2)+power(o1.cy-@cy,2)+power(o1.cz-@cz,2)))) < @r -- careful filter on distance Bounding box Eliminates the ~ 21% = 1-π/4 False positives

45. Summary • SQL is a set oriented language • You can express constraints as rows • Then You • Can evaluate LOTS of predicates per second • Can do set algebra on the predicates. • Benefits from SQL parallelism • SQL == Prolog? 

46. Talking the Talk and Walking the Walk • We have 7 people – what’s our agenda? • Observe it is all about:Data → Information → Knowledge → WisdomPeople == Communication is the “killer app” • ½ Personal Information Management • ½ Corporate Information Management