1 / 110

Spatial Business Detection and Recognition from Images

Spatial Business Detection and Recognition from Images. Alexander Darino. Outline. Project Overview. Previous Work Project Objective Anticipated End Result Project Pipeline. Project Overview. Previous Work: Where Am I?. Image. Where Am I?. Latitude, Longitude. Project Objective.

gada
Télécharger la présentation

Spatial Business Detection and Recognition from Images

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Spatial Business Detection and Recognition from Images Alexander Darino

  2. Outline • Project Overview

  3. Previous Work Project Objective Anticipated End Result Project Pipeline Project Overview

  4. Previous Work: Where Am I? Image Where Am I? Latitude, Longitude

  5. Project Objective • Given: • Image • Geolocation • Yield: • Spatial Identification of Businesses in Image • Addresses of Businesses in Image • Information about Businesses in Image • Ex. Reviews, Categories, Phone Number, etc.

  6. Project Objective • Given: • Image • Geolocation • Yield: • Spatial Identification of Businesses in Image • Addresses of Businesses in Image • Information about Businesses in Image • Ex. Reviews, Categories, Phone Number, etc.

  7. Project Pipeline Business Name Matching Business Spatial Detection Latitude Longitude Geocoding Reverse Geocoding Nearby Businesses BusinessIdentification Image Text Extraction Detected Text

  8. Anticipated End Result

  9. Obtaining a List of Candidate Businesses in Image via Business Searching

  10. Business Searching Business Name Matching Business Spatial Detection Latitude Longitude Geocoding Reverse Geocoding Nearby Businesses BusinessIdentification Image Text Extraction Detected Text

  11. Business Searching • Business Search Services • Google • Yelp • CityGrid (Supplier for Yellow Pages, Super Pages) • REST-based API • Results in JSON or XML format • Aggregate Results into Facade

  12. {'businesses': [{'address1': '466 Haight St',                 'address2': '',                 'address3': '',                 'avg_rating': 4.0,                 'categories': [{'category_filter': 'danceclubs',                                 'name': 'Dance Clubs',                                 'search_url': 'http://yelp.com/search?find_loc=466+Haight+St%2C+San+Francisco%2C+CA&cflt=danceclubs'},                                {'category_filter': 'lounges',                                 'name': 'Lounges',                                 'search_url': 'http://yelp.com/search?find_loc=466+Haight+St%2C+San+Francisco%2C+CA&cflt=lounges'},                                {'category_filter': 'tradamerican',                                 'name': 'American (Traditional)',                                 'search_url': 'http://yelp.com/search?find_loc=466+Haight+St%2C+San+Francisco%2C+CA&cflt=tradamerican'}],                 'city': 'San Francisco',                 'distance': 1.8780401945114136,                 'id': 'yyqwqfgn1ZmbQYNbl7s5sQ',                 'is_closed': False,                 'latitude': 37.772201000000003,                 'longitude': -122.42992599999999,                 'mobile_url': 'http://mobile.yelp.com/biz/yyqwqfgn1ZmbQYNbl7s5sQ',                 'name': 'Nickies',                 'nearby_url': 'http://yelp.com/search?find_loc=466+Haight+St%2C+San+Francisco%2C+CA',                 'neighborhoods': [{'name': 'Hayes Valley',                                    'url': 'http://yelp.com/search?find_loc=Hayes+Valley%

  13. Business Searching: Results 40.441127247181797 -80.002821624487595 Denham & Company Salon Ullrich's Shoe Repairing Nicholas Coffee Co Bella Sera On the Square A & J Ribs Starbucks Coffee Jenny Lee Bakery Galardi's 30 Minute Cleaners Jimmy John's Gourmet Sandwiches Charley's Grilled Subs Fresh Corner Lagondola Pizzeria & Restaurant Camera Repair Service Inc Pittsburgh Cigar Bar Original Oyster House MixStirs 1902 Tavern Costanzo's Pittsburgh Silver Llc Graeme St Galardi's 30 Minute Cleaners Denham & Co Salon Bruegger's Bagel Bakery Nicholas Coffee Co Market Square Fat Tommy's Pizzeria Mixstirs Cafe Giggles Rycon Construction Inc Garbera, Dennis C, Dds - Emmert Dental Assoc Bella Sera on the Square Mancini's Bread Co Las Velas Ciao Baby Washington Reprographics Inc Highmark Life Insurance Co Fischer, Donald R, Md - Highmark Life Insurance Co Jimmy John's Lynx Energy Partners Inc Emmert Dental Assoc

  14. Business Searching: Evaluation • Strengths • Aggregated results almost always found Business of interest • Weaknesses • Each API limits query result set size - this is why we aggregate • Only businesses listed • Not all businesses listed • Limitations • Dependent on well-populated, accurate Business Directories • Have only tested for 15 Pittsburgh images - unknown result quality for rural areas.

  15. Obtaining names of Businesses in Image by Extracting Identifying Text

  16. Extracting Identifying Text Business Name Matching Business Spatial Detection Latitude Longitude Geocoding Reverse Geocoding Nearby Businesses BusinessIdentification Image Text Extraction Detected Text

  17. Extracting Identifying Text: OCR • Used Two OCR APIs: • GNU OCR (Ocrad) • GOCR • OCR APIs highly sensitive to: • Font (only works well with roman font) • Perspective • Scale • Binarization Threshold • Dark on Light vs. Light on Dark (inversion)

  18. Extracting Identifying Text: OCR • OCR API evaluations • Ocrad - could not yield any meaningful data across over 200 scale/threshold/inversion combinations • GOCR - produced good results across 10 scales with and without inversion using threshold automatically determined by Otsu's method • 98% of Results are garbage! • Examples of GOCR output (next slides)

  19. Extracting Identifying Text: OCR

  20. Extracting Identifying Text: OCR n.c.......o.a...u..............oU..D.oa..e......_RuEGGE..KERy..J...w...........L........M.II.....c.. ...i.......l..J.t...llt...lSHA.P.It..tllt.........._.l...Jy._.c_...._tt.._....t.._.r.........t.t_t.._.._.l..J.r.r.I.

  21. Extracting Identifying Text: OCR

  22. Extracting Identifying Text: OCR u..........._nq......eoR.E.l.e...í....e...n... .n....n.e.R.E...e....o. _....E.R.E.IKE........I.ltlO.. .......rE..o......E.....I.K.E.o..... J.n....c...E.R.E.I.E...... .M..E.R.E...E...a J...Gu.ge..geE.F .._.....E..gE.D... fUlI..lll.lll.IIi.l..Xl..

  23. Extracting Identifying Text: OCR

  24. Extracting Identifying Text: OCR ..e_..w.._......D.........uJ.....J.................n......n..........n_..r.l_d..J.ec.m._..n.......J.n.._...tn..ct..._.................D.u.v...e.n....u.. Y.._w.n.n....Jn.......G..o..r..._........J...ml.t..l.tt.l.._w....................._....l....t........j..ilI.i..

  25. Extracting Identifying Text: OCR

  26. Extracting Identifying Text: OCR __.ncu_.l..._..._J...ne......._n._..v.....ra......d_..._.............i..n..UllREsT.unAN...r.c.....r...Tt.rJll......m...c.....n....... ...Jn.I..c...r.rESTAU.ANT.r.O....c.cc. Note: Even though "Tambellini" is a roman font, it is too stretched to be picked up by GOCR

  27. OCR Evaluation • Strengths • Applicable to expected input of orthogonal images • Weaknesses • Only works well(-ish) for strictly roman font • Limitations • Will perform poorly for artistic fonts and business signs • Conclusion • By itself, OCR is not the best approach towards Business identification • Reasons: poor recognition, franchises, perspective, etc

  28. Matching Identifying Text to Candidate Business Names via Business name Matching

  29. Business Name Matching Business Name Matching Business Spatial Detection Latitude Longitude Geocoding Reverse Geocoding Nearby Businesses BusinessIdentification Image Text Extraction Detected Text

  30. Business Name Matching • Given: Unreliable fragments of ‘detected text’ • Yield: Matching Business Names • Process: • Filter input: trimming, uselessness (< 2 letters) • Fuzzy String Matching • Voting Scheme: confidence of business appearing in image

  31. Business Name Detection

  32. Business Name Matching • Developed Confidence Attribution Algorithm • Confidence of OCR Token being Name Token • Example: Confidence of “ESTUANT” representing “RESTAURANT” • Point-based system • Confidence of Name appearing in Image • Sum of points of matching OCR Text • Use logarithmically-normalized points to determine business inclusion threshold

  33. Business Name Matching

  34. Business Name Matching

  35. Business Name Matching

  36. Business Name Matching

  37. Business Name Matching Note: This originally did not appear because it did not exceed the confidence threshold. It now appears because it contributes to the Business Name Identification

  38. Isolating Identified Images in Image via Spatial Business Identification

  39. Business Spatial Identification Business Name Matching Business Spatial Detection Latitude Longitude Geocoding Reverse Geocoding Nearby Businesses BusinessIdentification Image OCR Detected Text

  40. Business Spatial Identification

  41. Business Spatial Identification Aiken George S Co Category: Food, Grocery Address: 218 Forbes Ave Pittsburgh, PA 15222 Phone: (412) 391-6358 Rating: 4.5/5 (2 Reviews)

  42. Business Spatial Identification

  43. Business Spatial Identification

  44. Business Spatial Identification Bruegger's Bagels Category: Bagels Address:Market Sq Pittsburgh, PA 15222 Phone: (412) 281-2515 Rating: Not Rated

  45. V0.1: Evaluation

  46. Current Approach Business Name Matching Business Spatial Detection Latitude Longitude Geocoding Reverse Geocoding Nearby Businesses BusinessIdentification Image OCR Detected Text

  47. Weaknesses to Current Approach Business Name Matching Business Spatial Detection Latitude Longitude Geocoding Reverse Geocoding Nearby Businesses BusinessIdentification Image OCR Detected Text

  48. Weaknesses to Current Approach Lots of Garbage

More Related