1 / 15

Nikos Manolis Agro-Know Technologies

Tutorial on data aggregation and accessing datasets. Nikos Manolis Agro-Know Technologies. There is a lot of data. Need for data aggregation and harmonization. Objectives. This presentation aims to provide information on: How to use a service for aggregating datasets

bonner
Télécharger la présentation

Nikos Manolis Agro-Know Technologies

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Tutorial on data aggregation and accessing datasets Nikos Manolis Agro-Know Technologies

  2. There is a lot of data

  3. Need for data aggregation and harmonization

  4. Objectives This presentation aims to provide information on: • How to use a service for aggregating datasets • How to get already processed datasets • How to search processed datasets with a search API • Educational – GLN API (21008 res) • Bibliographic – ABN API (451602 res)

  5. The agDataHarvester service • Implements the OAI-PMH protocol to harvest metadata records from open data providers • REST-based API • Harvested dataset available through HTTP

  6. agDataHarvester parameters { "document_type": "harvesting_target", "harvesting_target": { "name":"Repository name", "description":”ShortRepositoryDescription", "url":"OAI-PMH target URL", "type":"metadata format prefix", "frequency":hours } }

  7. param.json { "document_type": "harvesting_target", "harvesting_target": { "name":"Indian Academy of Science", "description":"Indian Academy of Science", "url":"http://repository.ias.ac.in/cgi/oai2", "type":"mets", "frequency":24 } } • curl -X POST -d@param.json tp://'demo001':aginfra@agro.ipb.ac.rs/agcouchdb • { "ok": true, "id": " 5c56a3fa18fa21d2a85fd63cc9eb78ac ", "rev": "1-19ef1210376df8f1695a32b53ecb963a" } curl -X POST -d@INDUS.json http://'demo001':aginfra@agro.ipb.ac.rs/agcouchdb

  8. Get details on the dataset http://agro.ipb.ac.rs/agcouchdb/_design/datasets/_list/search/list?dataset.process_parameter_id=5c56a3fa18fa21d2a85fd63cc9eb78ac

  9. Get details on the dataset http://agro.ipb.ac.rs/agcouchdb/_design/datasets/_view/list_by_process?key=agdataharvester {"id": "6796259b52d79e4797e210c06e6a0aee", "key": "6796259b52d79e4797e210c06e6a0aee", "value": { "_id": "6796259b52d79e4797e210c06e6a0aee", "_rev": "1-d55d7bc90d26db64dae328c9328e4e4a", "document_type": "harvesting_target", "harvesting_target": { "name": “WorldBank", "description": "The World Bank - Open Knowledge Repository", "url": ""https://openknowledge.worldbank.org/oai/request", "type": “mets", "frequency": 24 }, "document_publisher": { "address": "83.212.96.169", "author": "demo001", "utc_datetime": "Wed Dec 11 11:58:45 2013", "utc_timestamp": 1386763125 } } }

  10. The agWorkflow service I wantall datasets with educational resources processed by the agINFRA powered aggregation workflow ! http://agro.ipb.ac.rs/agcouchdb/_design/datasets/_list/search/list? dataset.process=agworkflow&dataset.type=oai_lom&dataset.accuracy=true I wantall datasets with bibliographic resources processed by the agINFRA powered aggregation workflow ! http://agro.ipb.ac.rs/agcouchdb/_design/datasets/_list/search/list? dataset.process=agworkflow&dataset.type=oai_agris&dataset.accuracy=true

  11. Is there a way to search on available datasets ?

  12. Search API • REST-based queries over harmonized information (resultof metadata processing) • Two data models supported • akif: describing educational resources for agriculture, http://54.228.180.124:8080/search-api/v1/akif/?q=* • agrif: describing bibliographic resources for agriculture (mainly from FAO’s data), http://212.189.145.241:8080/search-api/v1/agrif/?q=*

  13. Search options • Simple search http://BASE_URL/search-api/v1/akif/?q=tomato • Searching within specific fields http://BASE_URL/search-api/v1/akif/?languageBlocks.en.description=tomato • Temporal http://BASE_URL/search-api/v1/akif/?creationDate=2013-04-16 • Fetching specific items http://BASE_URL/search-api/v1/akif/COLLECTION/20296

  14. Managing results • Sorting results e.g ?q=*&sort_by=creationDate&sort_order=desc • Facets e.g ?facets=set&facet_size=3 • Pagination e.g?q=sea&page_size=25&page=3 Full Documentation : 54.228.180.124:8080/search-api/

  15. Nikos Manolis Agro-Know Technologies manolisn@agroknow.gr

More Related