Evaluation of Dynamic Bioinformatics Services

1. Tenth International Workshop CIA 2006, Edinburgh Evaluating Dynamic Services in Bioinformatics Maíra R. Rodrigues Michael Luck University of Southampton, UK

2. Outline • Bioinformatics • Agents and Bioinformatics • Model for Cooperative Interactions: Overview • Requirements for Service Evaluation • Evaluation Method • Example Scenario • Conclusion • Future Work ECS - University of Southampton

3. Bioinformatics • Bioinformatics • Application of computer technology to manage and analyse biological data • Bioinformatics Services • Heterogeneous • Locally and remotely used • Continuous update • Management and analysis of biological data and tools • Suitability of an agent-based approach ECS - University of Southampton

4. Bioinformatics • Interrelated data • Cooperative applications • Participants request and provide services to each other • Services free of charge • Non-economic exchange of different types of tools and data • Interactions are based on reciprocal relations ECS - University of Southampton

5. Agents and Bioinformatics • The agent-based approach: • Agents provide and request bioinformatics services • Existence of alternative providers • Services are provided with different levels of quality (heterogeneity) • Therefore.. • Agents need to select service providers ECS - University of Southampton

6. Agents for Interaction • Agent-based applications in bioinformatics: • Concerned with high-level management tasks • Our concern: • Model non-economic cooperative interactions • Evaluation method for bioinformatics services to determine an agent’s satisfaction • Guide agent’s decisions over service providers ECS - University of Southampton

7. Model for Cooperative Interactions • Model non-economic cooperative interactions based on exchange values (Piaget 1973) • effort • satisfaction A1 A2 service • credit • debt ECS - University of Southampton

8. Model for Cooperative Interactions • Model non-economic cooperative interactions based on exchange values (Piaget 1973) • effort • satisfaction A1 A2 • credit • debt • debt • credit A1 A2 • effort • satisfaction ECS - University of Southampton

9. Model for Cooperative Interactions • Exchange values result from the agent’s evaluation of the service Service Evaluation Exchange Values Partner Selection (future interactions) ECS - University of Southampton

10. Model for Cooperative Interactions • Exchange values result from the agent’s evaluation of the service • Exchange values (Rodrigues, Luck 2005, 2006) • Current work focus on service evaluation Service Evaluation Exchange Values Partner Selection (future interactions) ECS - University of Southampton

11. Service Evaluation • Bio-Services are dynamic: • Constant updates • Regular behaviour, but • Sensitive to different parameter configuration • Evaluation requires • Repeated evaluation • Attach context information • Evaluation of different aspects of the service ECS - University of Southampton

12. Service Evaluation • Evaluation method should address: • Generality: apply to different types of bio-services and aspects of these services • Continuity: repeat evaluation every time a service is received • Consistency: compare evaluations made at different points in time • Discriminated information: allow flexible decision-making by using evaluation of individual aspects or a global evaluation ECS - University of Southampton

13. Alternative Approaches • Quantitative approaches • Scoring or utility functions • Objective values • Precision, consistency, combination is straightforward • Qualitative approaches: • Classification rules (e.g., poor, good, excellent) • Subjective values ECS - University of Southampton

14. Evaluation Method • Choose evaluation attributes for service • examples: performance, quality, reliability, etc. • For each attribute, associate result measures • Pieces of information derived from service result that can determine the service utility in relation to an attribute (observed value). • Static or dynamic measures (e.g., quality of interface and response time) ECS - University of Southampton

15. 1 Ui 0 c Evaluation Method • General evaluation function for evaluation attributes (utility): • For a set of attributes A = {a1,..,ai} • Ui = bc result measure for ai evaluation strictness ECS - University of Southampton

16. Evaluation Process • Before evaluation: • Identify evaluation attributes for services and result measures for each attribute • Repeat evaluation process every time a service is received • Input is the service result and configuration used • For each evaluation attribute ai • Compute result measures • Calculate evaluation Ui • Store evaluation • Output is a set of evaluations (evaluation tuple) ECS - University of Southampton

17. Evaluating Bio-Services • Proteomics research • Protein identification services • Input: file (list of unknown peptides) • Process: database + matching algorithm • Output: list of proteins, peptides per protein • Services: OMSSA, MASCOT, Tandem Local and Remote • Heterogeneous results for same input data • Sensitive to different input configurations • Evaluation can be used as criterion for future selection ECS - University of Southampton

18. Evaluating Bio-Services • Evaluation attributes: • Sensitivity • Capacity of matching related proteins • Accuracy • Capacity of identifying true matches • Performance • Time taken from input submission until result is received ECS - University of Southampton

19. Evaluating Bio-Services • Result measures (rm): • Sensitivity • Number of proteins • Peptide ratio - peptides per protein • Influence of input size • Increasing utility input_size peptide_ratio x protein_number rm = ECS - University of Southampton

20. Evaluating Bio-Services • Accuracy • Number of false positives • Decreasing utility rm = false_positives • Performance: • Response time • Influence of input size • Decreasing utility response_time input_size rm = ECS - University of Southampton

21. Evaluating Bio-Services • Evaluation functions: • Ui = 0.5rm • Sensitivity (U1): • U1 increases with peptide_ratio and protein_number • Accuracy (U2): • U2 decreases with false_positives • Performance (U3): • U3 decreases with response_time ECS - University of Southampton

22. Evaluating Bio-Services • Practical evaluation: • Same input spectra • Two different configurations (C1 and C2) • Evaluation of sensitivity • Evaluation reflects different results for C1 and C2 ECS - University of Southampton

23. Evaluating Bio-Services • Evaluation of performance • Again, evaluation reflects different results for C1 and C2 ECS - University of Southampton

24. Conclusions • Present an evaluation method to be used by agents requesting dynamic services in bioinformatics • Discussion of issues for efficient evaluation of these services, including • Adoption of a repeated evaluation process • Absolute evaluations • Generation of individual and compatible evaluations • Single evaluation must be calculated during selection ECS - University of Southampton

25. Conclusions • Show the application of the evaluation method for protein identification services • Importance of dynamic (repeated) evaluation is shown through empirical results • Provide more accurate information for agents that need to select services with dynamic characteristics ECS - University of Southampton

26. Future Work • Develop selection strategies that use and combine service evaluations • Combination through objective and subjective values • Probabilistic analysis of past evaluations • Consider similarity between different service configurations • Validate evaluation results with those of bioinformaticians ECS - University of Southampton

27. Thank you ECS - University of Southampton

28. References • J. Piaget. Sociological Studies. Routlege, London, 1973. • M. R. Rodrigues and M. Luck. Analysing partner selection through exchange values. In Jaime Sichman and Luis Antunes, editors, Multi-Agent-Based Simulation VI, volume 3891 of Lecture Notes in Artificial Intelligence, pages 24-40, Berlin Heidelberg, 2006a. Springer-Verlag. • M. R. Rodrigues and M. Luck. Cooperative interactions: An exchange values model. In Coordination, Organization, Institutions and Norms in Agent Systems (COIN), ECAI Conference, Riva del Garda, Italy, August 2006b. ECS - University of Southampton