160 likes | 261 Vues
Learn about assessing ontologies in BioPortal, ontology metrics, user reviews, peer review, and metadata for ontology versioning. Discover how to choose appropriate ontologies for your projects.
E N D
Ontology Evaluation, Metrics, and Metadata in NCBO BioPortal Natasha Noy Stanford University
What Are We Trying to Do? • You’ve built an ontology, how do you let the world know? • You need an ontology, where do you go to get it? • How do you know whether an ontology is any good?
BioPortal: A Community-Based Ontology Repository http://bioportal.bioontology.org
BioPortal Today • 195 ontologies in biomedical domain (as of 4/28/2010) • Ontologies in six different formats • ~ 1.7 million classes • ~ 2 million mappings
Evaluating Ontologies in BioPortal • Ontology metrics • Computed automatically • User reviews • Submitted by users: star ratings and free-text reviews along different dimensions • Other metadata • “Stamp of approval” from a specific community
Ontology Metrics • You can find the details here:http://www.bioontology.org/wiki/index.php/Ontology_Metrics • Statistics: • Number of classes • Number of instances • Number of properties • Maximum depth of the class hierarchy • Maximum number of siblings • Average number of siblings • Metrics assessing conformance to best practices: • Classes with a single subclass • Classes with a large number of subclasses (more than 20) • Classes with no definition • Auditing information: • Contributors and authors of individual classes
The Metrics Page in BioPortal You can view metrics for any ontology
Peer Review of Ontologies Which ontology is appropriate for my task? • The people who know the answer to these question are • (maybe) ontology authors • other users of the ontology • Our solution: Allow users to provide ratings for ontologies
Reviewers Provide • General review and rating • Usage information • Which applications have successfully used the ontology? • What problems were encountered? • Coverage • Does it cover the domain properly? • Are there major gaps? • Are some parts developed better than others?
Projects in BioPortal • Users can describe their ontology-based projects in BioPortal • Projects are linked to ontologies they are using • Ontology reviews can be done in the context of projects
Dimensions of a Review • Degree of formality • Documentation and support • Usability • Domain coverage • Correctness • Quality of content
Organizational Stamps of Approval • BioPortal ontologies belong to different groups: • OBO Foundry members and candidates • WHO Family of International Classifications • UMLS • caBIG • We can add other groups…. • Each ontology can be in 0, 1, or more groups • We should (but currently don’t) link to the criteria for inclusion in each group
Maintaining Metadata throughOntology Versioning • We attach reviews to a specific version of an ontology • We expose the reviews for all subsequent versions • We plan to add an indicator “This review was submitted for an earlier version of this ontology” Potentially, any part of the description can change
Representing Metrics and Reviews • Ontology Metadata Vocabulary (OMV): • OMV is a metadata schema that captures salient information about an ontology • OMV is implemented as an OWL ontology • BioPortal Metadata ontology extends OMV to add NCBO-specific metadata and evaluation metrics
Discussion • We can extend the BioPortal Metadata ontology to represent • Ontology profiles • Additional metrics • Sets of metrics and criteria for a specific organization • All information in BioPortal is accessible through REST services • Other applications can use the metadata • For example, there can be an external application that compares different profiles • All metadata is specific to an ontology version, but it is accessible through a version-independent ontology id