Gilbert Maiga and Ddembe Williams.A Flexible Approach for User Evaluation of Biomedical Ontologies.International Journal of Computing and ICT Research, Vol. 2, No. 2, pp. 62-74, December 2008.
Available at http://www.ijcir.org/volume2-number2/article6.pdf
The author discussed the existing approaches to ontology evaluation and conducted a field study to validate user requirements for evaluating a biomedical ontology based on structured interviews.
The author classified the taxonomy of evaluation approaches into 4 categories according to literatures:
(1)Gold standard approaches that compare an ontology to a gold standard;
(2)Task based approaches that assess results after using the ontology in an application;
(3)Data or corpus driven approaches that compare the fit of an ontology to domain texts;
(4)Assessment by humans to show how well the ontology meets a set of predefined criteria;
The author investigated 580 doctors and 50 biologists to test requirements for the evaluation framework of an ontology. The requirements include:
(2)Relevance of concepts
(3)Compare to Meta ontology
(4)Determine fit between ontology and task
(8)Wide Scope (Generic)
(10)Model aggregation (granularity)
(11)Reference biomedical model
The results showed that most respondents agree with the 11 requirements of evaluation framework above. In terms of visualize structure, there was 91% respondents stating that they agreed with it.
Moreover, the author constructed a user centered biomedical ontology evaluation framework. The figure is as follows: