1 / 23

Using Transliteration with Entity Resolution for Arabic Datasets

Using Transliteration with Entity Resolution for Arabic Datasets. Marwah Alian Ghazi Al- Naymat Banda Ramadan. Real Time Entity Resolution. Entity Resolution : is the process of Identifying records that represents the same real world entity. Real world entity could be:

gstrickland
Télécharger la présentation

Using Transliteration with Entity Resolution for Arabic Datasets

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Using Transliteration with Entity Resolution for Arabic Datasets MarwahAlian Ghazi Al-Naymat Banda Ramadan

  2. RealTime Entity Resolution • Entity Resolution : • is the process of Identifying records that represents the same real world entity. • Real world entity could be: • a person, a product, or a business or any other object in the real world.

  3. The number of comparisons required for query-based matching in the real-time ER process

  4. Examples of duplicates • A patient who is represented several times in a hospital DB • A product that is inserted many times in an inventory list.

  5. Dynamic Similarity-Aware Inverted Indexing (DySimII) • DySimII is a real-time ER technique that works with dynamic databases • It aims at providing real-time entity resolution for a stream of query records. • To use this approach, three indexes are needed

  6. DySimII steps • Step 1: Data Preprocessing • Step 2: Indexing • Step 3: Record Pair Comparison • Step 4: Classification • Step 5: Evaluation

  7. Data Preprocessing • Cleaning the data and make it standard to be used. Cleaning Data ready to be used Initial Data

  8. Indexing • Creates candidate records that potentially corresponds to matches. • Values are added to indexes whenever a query is being processed.

  9. Inverted Index • an inverted index is an index data structure storing a mapping from content such as words or numbers to its locations in a database file, or in a document or a set of documents.

  10. Inverted Indexes • Three Indexes are used in DySimII • Block Index (BI) • Similarity Index (SI) • Record Index (RI)

  11. Block Index (BI) • An inverted index that stores unique attribute values and their blocking key values. • Blocking key is generated using encoding function (phonics or soundex)

  12. Similarity Index (SI) • An inverted index that stores pre-calculated similarities between attribute values that are in the same block. • Keys in SI are unique attribute values, each key point to a list of precalculated similarities between this value and all other values in the same block.

  13. Record Index (RI) • Stores all unique attribute values and their associated record identifier. • Keys in RI are unique attribute values where each key points to a list of all record Identifiers that have the same attribute value.

  14. DySimII indexing Example • RI – record index • BI – blocking index • SI – similarity index Ramadan, B., Christen, P., Liang, H., Gayler, R., Hawking, D.: Dynamic similarity-aware inverted indexing for real-time entity resolution. (DMApps’13), Australia (2013)

  15. DySimII With Arabic Data • Several indexing techniques are used for Arabic unstructured data (texts and documents) • But there is no indexing technique used for arabic entity resolution. • In this work we apply DySimII on Arabic data, experiment its efficiency on Arabic ER.

  16. Block Index (BI) • This Index uses phonetics for names or words to be the blocking indexing key value • For Arabic data we use Transliteration before passing words to the soundex function

  17. Similarity Index (SI) SI 0.89 محمد محمود 0.6 أحمد ….. 0.4 سليمان سالم أحمد 0.89 محمود 0.6 محمد

  18. Record Index (RI) محمد أحمد سالم ….. RI r2 r10 r4 r6 r120 r200

  19. Record Pair Comparison محمد أحمد سالم سليمان محمد محمود 0.4 0.89 1 2.29

  20. Evaluation • Entity resolution process is evaluated using a variety of measures. • Using Two metrics: • Matching accuracy. • Average MRR (mean reciprocal rank):.

  21. Datapreparation • We use GeCo corruptor to make the deduplication with corrupted data. • GeCo is updated to be suitable for arabic dataset. • We use four corruptors: • Missing ValueCorruptor • Corrupt Value OCR:depends on similar characters • Keyboard corruptor • Edit Value Corruptor: selects randomly edit operation (insert, delete, substitute, or transpose)

  22. Results Specially with more corruption in data

  23. Conclusion • The results showed that to be able to use DysimII with Arabic scripts it is important to apply the transliteration step before building the indexes. These indexes were then used successfully in the ER process. • The results of using translitration were compared with the use of the stem as a blocking key index and It outperform the results of the stem.

More Related