Open Access

Developing a knowledge base to support the annotation of ultrasound images of ectopic pregnancy

  • Ferdinand Dhombres1Email author,
  • Paul Maurice1,
  • Stéphanie Friszer1,
  • Lucie Guilbaud1,
  • Nathalie Lelong2,
  • Babak Khoshnood2,
  • Jean Charlet3,
  • Nicolas Perrot4,
  • Eric Jauniaux5,
  • Davor Jurkovic6 and
  • Jean-Marie Jouannic1
Journal of Biomedical Semantics20178:4

DOI: 10.1186/s13326-017-0117-1

Received: 7 June 2016

Accepted: 18 January 2017

Published: 31 January 2017



Ectopic pregnancy is a frequent early complication of pregnancy associated with significant rates of morbidly and mortality. The positive diagnosis of this condition is established through transvaginal ultrasound scanning. The timing of diagnosis depends on the operator expertise in identifying the signs of ectopic pregnancy, which varies dramatically among medical staff with heterogeneous training. Developing decision support systems in this context is expected to improve the identification of these signs and subsequently improve the quality of care. In this article, we present a new knowledge base for ectopic pregnancy, and we demonstrate its use on the annotation of clinical images.


The knowledge base is supported by an application ontology, which provides the taxonomy, the vocabulary and definitions for 24 types and 81 signs of ectopic pregnancy, 484 anatomical structures and 32 technical elements for image acquisition. The knowledge base provides a sign-centric model of the domain, with the relations of signs to ectopic pregnancy types, anatomical structures and the technical elements. The evaluation of the ontology and knowledge base demonstrated a positive feedback from a panel of 17 medical users. Leveraging these semantic resources, we developed an application for the annotation of ultrasound images. Using this application, 6 operators achieved a precision of 0.83 for the identification of signs in 208 ultrasound images corresponding to 35 clinical cases of ectopic pregnancy.


We developed a new ectopic pregnancy knowledge base for the annotation of ultrasound images. The use of this knowledge base for the annotation of ultrasound images of ectopic pregnancy showed promising results from the perspective of clinical decision support system development. Other gynecological disorders and fetal anomalies may benefit from our approach.


Application ontology Knowledge base Ectopic pregnancy


Ectopic pregnancy is a common early pregnancy complication

Ectopic pregnancy occurs in 1 to 2% of pregnancies in developed countries and is defined by the implantation of a gestational sac outside the endometrial cavity of the uterus [1, 2]. The direct mortality rate from ectopic pregnancy is estimated to be 16.9 per 100,000 ectopic pregnancies [2], and is responsible for 4 to 10% of pregnancy-related deaths around the world [3]. Fallopian tubes are the most common site for ectopics to implant (tubal ectopics) with about 95% of ectopic pregnancies located there. For the rest, the implantation occurs within the uterine wall, but outside the endometrial cavity. Non-tubal ectopics are more difficult to diagnose than tubal ectopics and are associated with a higher mortality and morbidity [4]. Delayed diagnosis is the main factor for ectopic pregnancy associated with maternal death [2] and also affects the success rate of future pregnancies [5].

Ectopic pregnancy diagnosis relies on ultrasound expertise

The positive diagnosis of ectopic pregnancy is established through ultrasound scanning. More specifically, transvaginal scanning has been demonstrated to be superior to transabdominal ultrasound [4]. Consistent with continuous improvement in imaging quality and expertise, it has been recently suggested that a skilled operator could achieve a definite diagnosis at the very first scan [6]. However, most hospitals still rely on a heterogeneous staff to manage patients at risk for ectopic pregnancy, including emergency physicians, sonographers, radiologists and/or doctors in training [2, 7], with different levels of training and expertise. Thus, three or more visits are needed for 50% of these patients [8].

A shared representation for ectopic pregnancy imaging

Existing repositories of medical terminologies and ontologies, namely the Open Biomedical Ontologies (OBO) Foundry [9], the National Center for Biomedical Ontology (NCBO) BioPortal [10] and the Unified Medical Language System (UMLS) Metathesaurus [11] do not include a comprehensive set of resources to represent ultrasound signs. None of the resources reviewed in a recent survey of biomedical imaging ontologies was suitable for ectopic pregnancy [12]. This domain involves concepts from various medical domains, namely medical imaging, human anatomy and obstetrics/gynecology (OB/GYN). While existing standard terminologies may support a formal and shared representation for parts of our domain, as do the Foundational Model of Anatomy (FMA) [13] and the Radiology Lexicon (RadLex) [14], none of them provides the appropriate granularity for ectopic pregnancy imaging. More precisely, RadLex supports the representation of signs from various imaging modalities (including 50 ultrasound imaging signs [15]), as well as their relations to various medical conditions (including ectopic pregnancy), which makes it the best resource for our domain. However, RadLex is insufficient, because there are no subclasses for “ectopic pregnancy” [RadLex:RID4942] and RadLex only provides two related signs (“ring of fire sign” [RadLex:RID35495] and “interstitial line sign” [RadLex:RID35308]), as illustrated in Fig. 1.
Fig. 1

Graphical view of the "ectopic pregnancy" concept from the Radiology Lexicon (RadLex, version 3.13.1), with a full expansion of the concepts neighborhood for "ectopic pregnancy"[RadLex:RID4942]


In this article, we present a new ectopic pregnancy knowledge base and its application to ultrasound image annotation. In this knowledge base, the signs of ectopic pregnancy are linked to specific types of ectopic pregnancy, the anatomical structures involved and the technical elements of imaging. We also developed an ontology to provide the vocabulary used in the knowledge base, as well as an application for annotating ultrasound images, which leverages the knowledge base. We demonstrate the use of the knowledge base on the annotation of clinical images.


In this section, we describe our approach to developing a knowledge base for ectopic pregnancy imaging. We start by describing the underlying ontology. We present the knowledge base. Finally, we describe the application developed to support the annotation of ectopic pregnancy ultrasound images. The overview of the ontology and knowledge base development is presented in Fig. 2.
Fig. 2

Overview of the design of the knowledge system for ectopic pregnancy: ontology design, reference image collection and application for image annotation

Ontology development

To build the ectopic pregnancy ontology (EPO), we acquired concepts from a medical corpus. We also reused concepts from existing terminologies. We organized these concepts into hierarchies.

Acquiring concepts from text

We extracted terms from a medical corpus and organized them into concepts.

Extracting terms from a medical corpus

In order to cover the terms for the features to be annotated on EP images (i.e., types of ectopic pregnancy image, imaging signs, anatomical locations and technical elements for ultrasound image acquisition), we used Natural Language Processing (NLP) techniques [16] to extract and select medical terms from a collection of medical texts from two sources, namely the medical literature and de-identified reports of ultrasound examinations. More specifically, we searched PubMed for all medical publications indexed with the MeSH term "Pregnancy, Ectopic" from January 2000 to December 2014 for which an abstract was available, resulting in a collection of 2795 abstracts. Additionally, we extracted 4260 de-identified ultrasound reports form the Early Pregnancy Clinic database at the University College London Hospital (UCLH), restricted to ectopic pregnancy cases from October 2006 to April 2014. The lexico-syntactic analysis of these texts was performed using the part-of-speech tagger TreeTagger [17] and the term extractor YaTeA ( A total of 40,237 single/multi-word candidate terms were extracted.

Organizing extracted terms into concepts

The appropriate vocabulary for ectopic pregnancy was developed from these candidate terms, using the platform for ontology development from text Terminae/DAFOE [16]. Two experts reviewed and selected candidate terms, and defined the relevant concepts for ectopic pregnancy image description. The experts followed general principles for ontology design (clarity, coherence, extensibility, minimal encoding bias, minimal ontological commitment) [1821].

Acquiring concepts from existing terminologies

Whenever possible, the experts reused elements from existing terminologies, following previously described methods [22, 23]. For example, fine-grained concepts for the description of the pelvic anatomy in the FMA (e.g., the uterus [FMAID:17558] and all its parts) were added to the ontology.

Organizing concepts into hierarchies

We organized the resulting concepts into a subsumption hierarchy and we added annotations and logical definitions to these concepts.

Organizing concepts into a subsumption hierarchy

We used a core ontology for the medical domain developed in our academic center (ontoMénélas) to support the interoperability with other resources in our organization [2429]. The subsumption hierarchy (i.e., is-a or subClassOf relations) was developed in a top-down approach [19, 30] leveraging expert knowledge in medical imaging and OB/GYN, and by reusing existing resources. In particular, we reused some of the subsumption relations from the FMA (among the FMA concepts that were added to the ontology) as previously described by the RadLex group [31].


All concepts for ultrasound signs of ectopic pregnancy were manually annotated. The minimal set of annotations included (i) one English label (ii) one textual definition in English (iii) one PubMed identifier (PMID) for the concepts extracted from the PubMed corpus. Other annotations were optional (e.g., synonyms and French version of the annotations). The mappings of anatomical concepts to FMA concepts were stored as annotations in the ontology. The FMA labels and definitions for these concepts were also added as annotations. All textual annotations were based on SKOS predicates (prefLabel, altLabel, definition) [32]. We used the biomedical ontology editor Protégé version 5 ( for editing the annotations.

Logical definitions

General concepts for categories of signs were defined in intension as opposed to extension. These concepts correspond to defined classes in the ontology. For example, the concept “color Doppler sign” denotes an imaging sign, visible during an ultrasound examination, using the color Doppler mode. Therefore, this concept is formalized with a logical definition leveraging the property “requiresMode” and the concept “color Doppler mode”. As a result, signs whose definition contains “requiresMode some color Doppler mode” would automatically be classified as subclasses of “color Doppler sign”.

Implementation in OWL

The ontology was represented using the Web Ontology Language, OWL [33]. The hierarchy was inferred with an OWL-DL reasoner (Hermit 1.3.8), which also checked the consistency of the ontology.

Knowledge base development

The ontology provides the vocabulary for describing ultrasound images of ectopic pregnancy, which we used for developing a sign-centric knowledge base to represent the relations of each sign to ectopic pregnancy types, anatomical structures and technical elements for the acquisition of ultrasound images. Technical elements include the “examination route”, the “examination mode”, and the “echographic view”. For example, the “ring of fire sign” concept is represented in Fig. 3 with its relations to a type of ectopic pregnancy (“tubal pregnancy” through the relation “epo:suggests”), to anatomical structures (“ampulla”, “tubal isthmus”, “frimbrial portion” through the relation “epo:hasLocation”) and to technical elements (“vaginal route”, “color Doppler mode (2D)”, “adnexal area view” through the relations “epo:requiresRoute”, “epo:requiresMode”, “epo:requiresView”, respectively).
Fig. 3

Simplified representation of the sign "ring of fire" in the knowledge base

This knowledge was asserted at the most general level and propagated through the subsumption hierarchies of the ontology. For example, although the concept “ring of fire sign” is not explicitly linked to “tubal pregnancy sign” in the knowledge base, this relation can be inferred from “ring of fire sign epo:suggests tubal pregnancy” and “tubal pregnancy sign owl:equivalentClass (epo:suggests some tubal pregnancy)”.

We assessed the domain and scope of the knowledge base using “competency questions”, the answers to which must be represented with relations (asserted or inferred) from the knowledge base [34]. Such questions included “what are the different implantation sites of ectopic pregnancies?”, “what are the imaging signs of cesarean section scar pregnancy?”, “which ultrasound mode is required to depict a ring-of-fire sign?”, and “what are the anatomical structures visible in an adnexal area view?”

Beside the development of the sign-centric knowledge base, we selected from the medical literature ultrasound images of ectopic pregnancies illustrating the signs represented in the knowledge base. We restricted the 2795 PubMed citations used for the text corpus to articles in English, indexed with the MeSH term “Ultrasonography” and for which the article was freely available. One of the authors (PM) selected relevant images from the articles, in which the ultrasound signs were precisely described and illustrated. He annotated the signs in the knowledge base with the PMID of the article. For example, the concept “ring of fire sign” is annotated with PMID 18936028 in reference to an article describing this sign [35].

Application development

We developed an application for the annotation of ultrasound images of ectopic pregnancy. This application leverages both definitional knowledge from the ontology and assertional knowledge from the knowledge base. The main features of this application include:
  1. i)

    searching for image annotations using terms from the ontology,

  2. ii)

    suggesting relevant signs based on the knowledge base, and

  3. iii)

    accessing reference images for a given sign.

The user interface was developed as a Java 7 web application based on open-source elements. The ontology and the knowledge base were stored in an RDF triple store (Apache Jena 3.0). We used queries against a SPARQL endpoint (Apache Fuseki) to access the knowledge base. Simple subsumption reasoning was sufficient to access all the asserted and inferred knowledge from the knowledge base. We established a set of SPARQL rules to suggest the signs, anatomical structures and technical elements associated with a given type of ectopic pregnancy selected by the user. For example, the following SPARQL query retrieves all ectopic pregnancy types having at least one sign from a given set of signs. “${selectedSigns}” is a variable containing the URIs of this set of signs, “${inferredGraph}” is the inferred ontology graph in the triplestore and "${language}" is the language used for label display in the system:
  • PREFIX owl:< #>

  • PREFIX rdf:< #>

  • PREFIX epo:< #>

  • PREFIX skos:< #>

  • PREFIX rdfs:< #>

  • SELECT DISTINCT ?disorder ?disorder_label ?disorder_definition

  • FROM ${inferredGraph}

  • WHERE {

  • VALUES ?sign {${selectedSigns}}

  • ?sign skos:hiddenLabel ?sign_id;

  • rdfs:subClassOf* epo:OPPIO_0000189 .

  • ?disorder skos:prefLabel ?disorder_label;

  • rdfs:subClassOf epo:OPPIO_c000016 .

  • OPTIONAL {?disorder skos:definition ?disorder_definition .}

  • ?sign rdfs:subClassOf* ?restr .

  • ?restr owl:onProperty epo:suggests .

  • ?restr owl:someValuesFrom/rdfs:subClassOf* ?disorder .

  • FILTER(lang(?disorder_label) = "${language}")

  • }

  • ORDER BY ?disorder_label

The result from this query is a list of ectopic pregnancy types (URI, label and definition) and can be used in subsequent queries to suggest new signs associated with these ectopic pregnancy types.


We conducted an evaluation of the ontology, the knowledge base and the application. The ontology and the knowledge base were evaluated through a questionnaire and users evaluated the application based on clinical cases.

Evaluation of the ontology: Does the ontology contain the appropriate vocabulary for ectopic pregnancy ultrasound imaging?

The vocabulary provided by the ontology was presented to a group of potential users with different levels of expertise. After a demonstration of the application followed by a brief hands-on session to search for terms, we collected feedback from each user by anonymous questionnaire. Questions assessed whether the terms for signs, anatomical structures, types of ectopic pregnancy and technical elements were consistent with their clinical practice and if they were able to find the signs they were looking for in the application.

Evaluation of the knowledge base: Are the suggested signs and images useful?

The signs and images suggested by the knowledge base were assessed by the same panel of users through another questionnaire. We asked users if they learned new signs for some types of ectopic pregnancy and whether the reference images provided were helpful for analyzing ultrasound images. Here we distinguished between junior and senior users, because our intuition was that the juniors are more likely than seasoned physicians to learn from our system.

Evaluation of the application based on clinical cases

Using our application, users annotated ultrasound images of ectopic pregnancy scans. This study was approved by the ethic committee of the French National College of Obstetrics and Gynecology (No CEROG 2015-GYN-1002). The ultrasound scans (reports and images) were randomly selected from ectopic pregnancy cases managed at the Pyramids Medical Imaging Center in Paris and the Early Pregnancy Unit at UCLH. All personally identifying information was removed from the text of the reports, from the content of the images and from the image metadata. Each observer was assigned a subset of 10 cases for analysis, of which 5 were common to all observers and 5 were specific. For each case, the observers were asked to annotate the images with the application. They were blind to the content of the ultrasound report.

Our motivation for this preliminary evaluation was not so much to assess whether all relevant signs had been annotated, but rather to ensure that the signs suggested by our application were appropriate. In other words, we focus on precision, not recall. Additionally, we evaluated the reproducibility of the annotations among the observers.


The gold standard for the presence of signs on each image was derived from the ultrasound reports provided by the specialist centers. We measured the precision of sign annotations provided by the observers (observed signs) against the signs from the gold standard (relevant signs). We used the usual definition for precision in information retrieval [36]:
$$ precision=\frac{\left|\left\{ relevant\ signs\right\}{\displaystyle \cap}\left\{ observed\ signs\right\}\right|}{\left|\left\{ observed\ signs\right\}\right|} $$


The measure for assessing the reproducibility of the annotations was the proportion of agreement for categorical assessment across multiple observers [37]. The proportion of agreement \( {p}_a \) for a given sign in a given image was the ratio of the number of agreements between the observers (i.e., the number of pairs of observers who agree) for the presence of the sign, over the number \( n \) of trials of agreement (i.e., the total number of pairs of observers). For example, considering a group of x = 6 observers, of whom 5 observers annotated one of the images with a given sign, the number of trials of agreement is n = 1 + 2 +  + (x − 1) = 15, and the number of agreements among the 5 observers is 10. Thus, p a is 10/15 = 66%. The 95% confidence interval (CI) for \( {p}_a \) was calculated from the Standard Error of the proportion: \( S E=\sqrt{p_a\left(1-{p}_a\right)/ n} \). Considering a standard normal distribution for p a , the 95% CI is p a  ± 1.96 × SE. Statistical computations were performed using R version 3.2 and STATA version 14.


Ectopic pregnancy ontology

As of June 2016, the ectopic pregnancy ontology (version 1.1) contains 1388 concepts to describe ectopic pregnancy ultrasound images, organized into several subsumption hierarchies for types of ectopic pregnancies and the signs, anatomical structures and technical elements of imaging associated with ectopic pregnancy. The usual metrics for ontology description are presented in Table 1. There are 24 classes for the types of ectopic pregnancy, as illustrated in Fig. 4. The 90 concepts for ultrasound signs include “endometrial trilaminar pattern”, “tubal ring sign”, and “ring of fire sign”. While most sign concepts are represented as primitive classes, some of them are defined classes. For example, the concept “color Doppler sign” is a defined class equivalent to [rdfs:subClassOfimaging sign” and epo:requiresMode some “color Doppler mode”]. In general, we created defined classes for the categories of signs by technical element (e.g., by the examination mode (e.g., “2D ultrasound sign”, “color Doppler sign”) and by implantation site of ectopics (e.g., “tubal pregnancy sign”, “c-section scar pregnancy sign”).
Table 1

Ectopic Pregnancy Ontology (v1.1) metrics

Class count


Object property count


Individual count


SubClassOf axioms count


EquivalentClasses axioms count


DisjointClasses axioms count


AnnotationAssertion axioms count

- skos:prefLabel


- skos:altLabel


- skos:definition


- epo:FMAID (FMA class UI)

Fig. 4

Taxonomy of ectopic pregnancy by implantation sites (view of the Ectopic Pregnancy Ontology)

There are 484 concepts for anatomical structures of the female pelvic anatomy (e.g., “uterus”, “uterine tube”, “zone of uterine tube” and “ampulla”) and early gestational structures (e.g., “gestational sac”, “trophoblast”). General anatomical concepts from the FMA were used to seed the hierarchy (e.g., “organ zone” and “non gestational anatomical structure”). Specialized concepts (e.g., “gestational sac”) were added to extend the FMA hierarchy as necessary for our application.

The technical element concepts were organized into three hierarchies for “examination route”, “examination mode” and “echographic view”. There are 3 examination route subclasses (e.g., “vaginal route”), 9 examination mode subclasses (e.g., “color Doppler mode”), and 17 echographic view subclasses (e.g., “longitudinal view of the uterus”).

The asserted subsumption hierarchy of the ontology involved 2707 relations. The domain and range of 44 relations (e.g., “hasLocation”, “suggests”, “requiresMode”) are defined in the ontology. Finally, this ontology includes no individuals, because instances of signs are the actual signs observed on images from a clinical case.

Knowledge base for image annotation

In the knowledge base, the 81 signs defined in the ontology are related to ectopic pregnancy types, anatomical structures and the three categories of technical elements (the echographic view, the examination mode and the examination route) as illustrated in Fig. 3. There are 169 asserted relations between these signs and the different types of ectopic pregnancy, as some signs can be associated with several types of ectopic pregnancy. Similarly, the signs can be related to multiple anatomical structures (with 239 asserted relations), as well as multiple technical elements (with 356 asserted relations). The asserted knowledge from the sign-centric knowledge base characteristics is summarized in Table 2. After inference in the knowledge base, 618 inferred relations between signs and types of ectopic pregnancy were produced, as well as 1503 inferred relations between signs and technical elements.
Table 2

Characteristics of the ectopic pregnancy knowledge base for imaging signs


Object property

Axioms (n)


- Ultrasound Sign ® Ectopic Pregnancy



- Ultrasound Sign ® Anatomical Structure



- Ultrasound Sign ® Technical Element




- PubMed citations



- Image from reference collection



The signs in the knowledge base are associated with reference images and PubMed citations. One hundred and six articles from 33 medical journals were reviewed for establishing the collection of reference images, resulting in the selection of 80 images depicting relevant ultrasound signs. A total of 77 PMID annotations and 98 image annotations illustrate the signs in the knowledge base.

Application for ultrasound image annotations

An overview of the user interface of the application is presented in Fig. 5. The image to annotate is displayed in the top left corner of the screen. The annotation search field at the bottom of the screen supports auto-completion for terms related to ectopic pregnancy types, anatomical locations, technical elements and ultrasound signs. The results are displayed in a sliding panel and the user can select the relevant terms, which are then added as image annotations in the top right corner of the screen as image annotation. As the user selects annotations, the system provides a selection of reference images from the collection in the bottom left corner to illustrate the selected annotations. Finally, in the bottom right corner, the system suggests other signs of interest based on the type of pregnancy, anatomical structure and technical elements selected.
Fig. 5

Overview of the interface of the web application for ectopic pregnancy ultrasound image annotations. This graphical user interface was developed using the AngularJS ( and Bootstrap (


Evaluation of the ontology: Does the ontology contain the appropriate vocabulary for ectopic pregnancy ultrasound imaging?

A total of 17 users (junior and senior OB/GYN practitioners and radiologists, and sonographers from France and the UK) were presented with the application. Their feedback on the terms available in the ontology was generally favorable. More specifically, 100% of the users found the vocabulary for the ectopic pregnancy signs to be consistent with their clinical practice, 94,1% for the anatomical structures and 82,4% for the terms describing technical elements of imaging. Moreover, 82.4% were able to find the signs they were looking for in the application, without further assistance.

Evaluation of the knowledge base: are the suggested signs and images useful?

Overall, half of the users (52.9%), including all five junior users, learned about new signs associated with ectopic pregnancy types. The reference images suggested by the application were “always” of “often” useful for 14 users (82.4%). One user considered that the suggested images were “sometimes” useful and two users considered the suggested images “rarely” useful. As expected, the usefulness of the application depended on the expertise of the user, with junior users benefitting most.

Evaluation of the application based on clinical cases

Six independent observers, all OB/GYN practitioners with different level of training in ultrasound imaging (three seniors, two senior registrars and one registrar) annotated 206 ultrasound images from 35 clinical cases of ectopic pregnancy (five common cases and five additional cases for each user). The cases are presented in Table 3. The observers provided 1486 annotations with an overall precision of 0.83. The precision for each sign is presented in Fig. 6.
Table 3

Types of ectopic pregnancy among the 35 ultrasound cases used for the annotation evaluation

Type of ectopic pregnancy

Cases in common

Other cases

Tubal pregnancy

3 (20 images)

12 (67 images)

Cesarean section scar pregnancy

2 (13 images)

12 (74 images)

Cervical pregnancy


4 (24 images)

Interstitial pregnancy


2 (8 images)


5 (33 images)

30 (173 images)
Fig. 6

Precision of ectopic pregnancy sign annotations in ultrasound images

For the five common cases, the observers used 46 distinct signs to create 841 annotations. For 783 annotations (covering 26 distinct signs), the annotation was made by at least two observers. The 58 remaining annotations (6.9%) were created by only one of the six observers and involved 20 distinct signs. The total proportion of agreement for the presence of signs in images was 40.35% [38.64%-42.05%]95%CI. The reproducibility for each sign annotation is presented in Table 4.
Table 4

Proportion of agreement on the presence of ultrasound signs in the 5 common cases of ectopic pregnancy

Sign annotations form the Ectopic Pregnancy Ontology

Agreement (n)

Lack of agreement (n)

Images (n)

Proportion of agreement (% and [95% CI])

Endometrial trilaminar pattern




80.00 [69.88 - 90.12]

Adnexal mass distinct from ovary




79.44 [73.54 - 85.34]

Embryo visible outside the uterine cavity




75.00 [64.04 - 85.96]

Gestational sac outside the uterine cavity




68.48 [63.47 - 73.49]

Adnexal mass adjacent to ovary




67.78 [60.95 - 74.61]

Gestational sac or trophoblast in a myometrial defect in previous caesarean section scar pregnancy site




61.03 [54.18 - 67.88]

Adnexal mass and corpus luteum at the same side




60.74 [52.50 - 68.98]

Adnexal rounded hyperechoic mass




52.50 [43.57 - 61.43]

Ring of fire sign




52.00 [40.69 - 63.31]

Yolk sac visible outside the uterine cavity




47.41 [38.99 - 55.83]

Adnexal mass as gestational sac with yolk sac




36.67 [24.48 - 48.86]

Caesarean section scar pregnancy peritrophoblastic blood flow




31.67 [19.90 - 43.44]

Intact endometrial midline echo




31.11 [17.58 - 44.64]

Tubal ring sign




27.27 [20.47 - 34.07]

Tubal ring without central identifying feature




26.67 [17.53 - 35.81]

Trophoblast visible outside the uterine cavity




25.29 [21.21 - 29.37]

Anterior distortion of uterus serosa




20.00 [12.84 - 27.16]

Smaller trophoblastic border distance to the anterior uterine serosa




16.92 [11.66 - 22.18]

Ectopic pregnancy wall more echogenic than corpus luteum wall




15.56 [9.45 - 21.67]

Fluid collection located centrally within the uterine cavity




13.33 [1.17 - 25.50]

Non intact endometrial midline echo




8.89 [0.57 - 17.21]

Gestational sac or trophoblast located at the level of internal cervical os




7.41 [2.99 - 11.83]

Gestational sac inside anterior myometrium and uterine cavity




2.22 [0.00 - 6.52]

Gestational sac located eccentricaly from uterine cavity




0.67 [0.00 - 1.98]



40.35 [38.64 - 42.05]


We have developed an application ontology, a knowledge base and an application for the annotation of ultrasound images of ectopic pregnancy. This was the first attempt to build semantic resources in this domain. We discuss the significance of our findings, as well as the limitations and perspectives of this resource from the perspective of clinical application development.


Using Sematic Web technologies [38] and ontologies [18], we successfully developed a comprehensive, unambiguous, shared and computable representation of the ectopic pregnancy ultrasound signs, for which existing resources were insufficient.

The ontology and the knowledge base received positive feedback from a panel of medical users (including mixed medical staff and sonographers). This preliminary evaluation demonstrates that they were able to identify morphological ultrasound features for a particular diagnosis and to associate them with pre-defined terms. The use of a large and diverse corpus as our source of vocabulary was critical for reaching a shared and fine-grained representation of the domain [20]. As expected, the signs described in the ontology are consistent with the most important signs for tubal pregnancy diagnosis identified in the recent meta-analysis by Richardson et al. [39].

The relevance of this application ontology is illustrated by a high precision rate of 83%, which reflects the proportion of correct sign annotations made by the observers. This result is especially encouraging at a time when we are considering using this knowledge base in a clinical decision support system.

The global proportion of agreement was 40.35%, which is satisfactory considering the number of images (33) and signs (26) involved. In comparison, a proportion of agreement of 50% was reported for the binary assessment of the abnormality of fetal heart rate in 20 cardiotocograms by 5 observers [37]. Interestingly, some signs with moderate proportions of agreement (e.g., “tubal ring sign”, p a = 27.3% [20.5-34.1]95%CI), had good precision rates (e.g., precision = .77 for “tubal ring sign”). Moreover, in these images, a more general sign with a higher agreement was present (e.g., “adnexal mass distinct from ovary”, p a  = 79.4% [73.5-85.3]95%CI). This was an expected effect of sign suggestions in the application.


Limitations of the ontology

Many biomedical ontologies developed recently have used the basic formal ontology (BFO) [40] as their top-level ontology for interoperability with other OBO ontologies. Instead, we used our local core ontology for medicine (Ménélas), because interoperability with other projects in our institution was more important. Moreover, the use of a top-level ontology was not a primary requirement in the design of our application ontology. Similarly, we did not use the popular MIREOT [41] strategy for referencing external resources in our ectopic pregnancy ontology. Because it was crucial for this application ontology to ensure the stability of our application, we decided to restrict to a minimum the ontological commitment that comes with the reuse of external, evolving ontologies. However, we kept the mapping to reference resources, such as the FMA.

Preliminary evaluation

In its current state, the application we developed only supports the annotation of clinical images, not the diagnosis of the conditions represented on these images. For our evaluation, most of the signs from the ontology used in annotations were tubal pregnancy signs, cesarean-section scar pregnancy signs and some signs that were not specific of a location. While sufficient for evaluating the precision and reproducibility of the annotations, this skewed dataset would be insufficient for the evaluation of a diagnostic system.

Toward a clinical decision support system (CDSS) for ectopic pregnancy diagnosis

There is a need for CDSS in the domain of ultrasound signs for early pregnancy. Except in specialist centers, many women with ectopic pregnancy will not be diagnosed by transvaginal ultrasound at their first visit. However, adequate management necessitates detailed ultrasound differential diagnosis of the different early pregnancy complications [4, 42] which requires advanced training [43]. In practice, only some of the initial transvaginal scans are performed by experts, thus delaying the appropriate diagnosis and treatment, increasing adverse outcomes and also generating a significant number of visits [8, 44, 45]. In this context, a CDSS for early identification of relevant ectopic pregnancy signs will likely benefit non-expert operators. However, developing a CDSS for early pregnancy is challenging for several reasons. Potential users have heterogeneous expertise; there is no standard terminology describing the relevant ultrasound signs; and the quality of ultrasound images varies significantly among operators.

We consider this ectopic pregnancy image annotation application, with its underlying ontology and knowledge base, a step toward a clinical decision system for ectopic pregnancy diagnosis. Research in CDSS based on ontologies has demonstrated differential diagnosis assistance in Human Genetics [46] or in conventional Radiology [47].

The precision of the annotations derived from our knowledge base is promising for developing a CDSS for ectopic pregnancy ultrasound. The prospective evaluation of a clinical decision support system (CDSS) based on our knowledge base should demonstrate improvement in clinical care. For example, the expectation would be that, junior operators guided by the signs suggested by the system achieve a better analysis of ultrasound images, and therefore reach the correct diagnosis more often than without the system. A specific challenge for such clinical evaluation is its integration in the clinical workflow.

Finally, the knowledge base we developed could be extended from ectopic pregnancy to early pregnancy (i.e., including molar pregnancy, miscarriage and multiple pregnancy at early stages of development), and more generally to the next stages of fetal development (i.e., to represent ultrasound signs associated with fetal disorders).


We have developed a new ectopic pregnancy knowledge base for the annotation of ultrasound images. The elements of this knowledge base (signs and types of ectopic pregnancy, anatomical structures involved and technical elements of imaging) are organized into an ontology. We have demonstrated the use of this knowledge base for the annotation of ultrasound images of ectopic pregnancy, with promising results from the perspective of clinical decision support system development. Other gynecological disorders and fetal abnormalities may benefit from our approach.



Clinical decision support system


Foundational model of anatomy


National center for biomedical ontology


Natural language processing


Open biomedical ontologies


Ontology web language


PubMed identification number


Radiology lexicon


Simple knowledge organization system


University College London Hospital


Unified Medical Language System



This work was conducted using the Protégé resource, which is supported by grant GM10331601 from the National Institute of General Medical Sciences of the United States National Institutes of Health. (website:

The lexicalization plugin Archonte 4.2 for Protégé 5, developed by Laurent Mazuel and supported by INSERM U1142 LIMICS, was used to edit the SKOS multilingual annotations in the ontology. (website:

The authors would like to thank Olivier Bodenreider for his expert advice and support throughout the final editing of this manuscript.


The development of the web-application was founded by the SATT-Lutech for the Pierre and Marie Curie University, Paris, France.

Availability of data and material

The ontology for ectopic pregnancy imaging is available on BioPortal:

Authors’ contributions

All the authors were involved in the drafting or the revising at different stages of the manuscript. FD and PM developed the ontology and knowledge base in collaboration with JC, JMJ, EJ and DJ. FD lead the development of the web application and the knowledge base integration. FD, PM, JMJ, and EJ established the evaluation protocols (questionnaires and clinical cases). PM presented the application and collected the questionnaires. DJ and NP, ectopic pregnancy experts, provided the de-identified clinical material (text, images). FD, NL, and BK did the statistical analysis. SF, LG tested the application at every stage of development. All authors read and approved the final manuscript.

Competing interests

The authors declared that they have no competing interest.

Consent for publication

Not applicable.

Ethics approval and consent to participate

This study was approved by the institutional review board of the French College of Obstetrics and Gynecology (No CEROG 2015-GYN-1002) for the use of de-identified human data.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

UPMC Medical Faculty (Paris 6), Department of Fetal Medicine in Armand Trousseau Hospital (APHP), INSERM U1142 (LIMICS)
INSERM U1153 (Obstetrical, Perinatal and Pediatric Epidemiology Research Team, Center for Biostatistics and Epidemiology)
Pyramides Medical Imaging Center
University College Hospital (UCLH) Department of Obstetrics and Gynaecology, Academic Department of Obstetrics and Gynaecology, University College London (UCL) Institute for Women’s Health
Department of Obstetrics and Gynaecology, Gynaecology Diagnostic and Outpatient Treatment Unit, University College Hospital (UCLH)


  1. Barnhart KT. Clinical practice. Ectopic pregnancy. N Engl J Med. 2009;361:379–87.View ArticleGoogle Scholar
  2. Cantwell R, Clutton-Brock T, Cooper G, Dawson A, Drife J, Garrod D, Harper A, Hulbert D, Lucas S, McClure J, et al. Saving Mothers' Lives: Reviewing maternal deaths to make motherhood safer: 2006-2008. The eighth report of the confidential enquiries into maternal deaths in the United Kingdom. BJOG. 2011;118 Suppl 1:1–203.Google Scholar
  3. Marion LL, Meeks GR. Ectopic pregnancy: History, incidence, epidemiology, and risk factors. Clin Obstet Gynecol. 2012;55:376–86.View ArticleGoogle Scholar
  4. Knez J, Day A, Jurkovic D. Ultrasound imaging in the management of bleeding and pain in early pregnancy. Best Pract Res Clin Obstet Gynaecol. 2014;28:621–36.View ArticleGoogle Scholar
  5. Ghaneie A, Grajo JR, Derr C, Kumm TR. Unusual ectopic pregnancies: sonographic findings and implications for management. J Ultrasound Med. 2015;34:951–62.View ArticleGoogle Scholar
  6. Jurkovic D, Wilkinson H. Diagnosis and management of ectopic pregnancy. BMJ. 2011;342:d3397.View ArticleGoogle Scholar
  7. Stein JC, Wang R, Adler N, Boscardin J, Jacoby VL, Won G, Goldstein R, Kohn MA. Emergency physician ultrasonography for evaluating patients at risk for ectopic pregnancy: a meta-analysis. Ann Emerg Med. 2010;56:674–83.View ArticleGoogle Scholar
  8. Wedderburn CJ, Warner P, Graham B, Duncan WC, Critchley HO, Horne AW. Economic evaluation of diagnosing and excluding ectopic pregnancy. Hum Reprod. 2010;25:328–33.View ArticleGoogle Scholar
  9. Smith B, Ashburner M, Rosse C, Bard J, Bug W, Ceusters W, Goldberg LJ, Eilbeck K, Ireland A, Mungall CJ, et al. The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration. Nat Biotechnol. 2007;25:1251–5.View ArticleGoogle Scholar
  10. Noy NF, Shah NH, Whetzel PL, Dai B, Dorf M, Griffith N, Jonquet C, Rubin DL, Storey MA, Chute CG, Musen MA. BioPortal: ontologies and integrated data resources at the click of a mouse. Nucleic Acids Res. 2009;37:W170–173.View ArticleGoogle Scholar
  11. Bodenreider O. The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic Acids Res. 2004;32:D267–270.View ArticleGoogle Scholar
  12. Smith B, Arabandi S, Brochhausen M, Calhoun M, Ciccarese P, Doyle S, Gibaud B, Goldberg I, Kahn Jr CE, Overton J, et al. Biomedical imaging ontologies: A survey and proposal for future work. J Pathol Inform. 2015;6:37.View ArticleGoogle Scholar
  13. Rosse C, Mejino Jr JL. A reference ontology for biomedical informatics: the Foundational Model of Anatomy. J Biomed Inform. 2003;36:478–500.View ArticleGoogle Scholar
  14. Rubin DL. Creating and curating a terminology for radiology: ontology modeling and analysis. J Digit Imaging. 2008;21:355–62.View ArticleGoogle Scholar
  15. Shore MW, Rubin DL, Kahn Jr CE. Integration of imaging signs into RadLex. J Digit Imaging. 2012;25:50–5.View ArticleGoogle Scholar
  16. Szulman S, Charlet J, Aussenac-Gilles N, Nazarenko A, Aussenac-Gilles N, Sardet E, Teguiak V: DAFOE: An ontology building platform from texts or thesauri. In Proceedings of the International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management. 2009
  17. Schmid H. Probabilistic Part-of-Speech Tagging Using Decision Trees. Manchester: Proceedings of International Conference on New Methods in Language Processing; 1994.Google Scholar
  18. Gruber TR. A translation approach to portable ontology specifications. Knowl Acquis. 1993;5:199–220.View ArticleGoogle Scholar
  19. Uschold M, Gruninger M. Ontologies - Principles, methods and applications. Knowl Eng Rev. 1996;11:93–136.View ArticleGoogle Scholar
  20. Bachimont B, Troncy R, Isaac A. Semantic Commitment for Designing Ontologies: A Proposal. In: Asuncion Gomez Perez, Benjamins R. Proc. 13th international conference on knowledge engineering and knowledge management (EKAW), Oct 2002. Lecture notes in computer science 2473. Siguenza: Springer Verlag; 2002. pp.114–21.
  21. Gruber TR. Toward principles for the design of ontologies used for knowledge sharing? Int J Hum Comput Stud. 1995;43:907–28.View ArticleGoogle Scholar
  22. Pinto HS, Martins JP. Ontologies: How can They be Built? Know Inf Sys. 2004;6:441–64.View ArticleGoogle Scholar
  23. Dhombres F, Charlet J, Jouannic J-M, Mazuel L. Re-use of terminological and ontological resources for the construction of domain ontologies in medicine: a description of two experimental approaches. EKAW 2010 Conference Workshop. Lisbonne: Springer; 2010.Google Scholar
  24. Baneyx A, Charlet J, Jaulent MC. Building an ontology of pulmonary diseases with natural language processing tools using textual corpora. Int J Med Inform. 2007;76:208–15.View ArticleGoogle Scholar
  25. Charlet J, Bachimont B, Jaulent MC. Building medical ontologies by terminology extraction from texts: an experiment for the intensive care units. Comput Biol Med. 2006;36:857–70.View ArticleGoogle Scholar
  26. Charlet J, Declerck G, Dhombres F, Gayet P, Miroux P, Vandenbuscche P-Y: Building a Medical Ontology to support Information Retrieval: Terminological and metamodelization issues. 2013. 2013
  27. Dhombres F, Aymé S, Rath A, Olry A, Vandenbussche P-Y, Charlet J: Automated generation of diseases classification supported by Orphanet Ontology of Rare Diseases. ICHG; 2011-10. 2011
  28. Dhombres F, Vandenbuscche P-Y, Rath A, Hanauer M, Olry A, Urbero B, Choquet R, Aimé X, Charlet J: The OntoOrpha Ontology - first propositions to organize Orphanet Knowledge in the Semantic Web paradigm. 2013. 2013
  29. Dhombres F, Vandenbussche P-Y, Rath A, Olry A, Hanauer M, Urbero B, Charlet J: OntoOrpha : an ontology to support edition and audit of rare diseases knowledge in Orphanet. In International Conference on Biomedical Ontology; 2011-07. Olivier Bodenreider, Maryann E. Martone, Alan Ruttenberg (eds.); 2011: 241-243.
  30. Noy N, McGuinness D. Ontology development 101: a guide to creating your first ontology. Knowl Systs Lab. 2001;32:1–25.Google Scholar
  31. Mejino Jr JLV, Rubin DL, Brinkley JF. FMA-RadLex - An Application Ontology of Radiological Anatomy derived from the Foundational Model of Anatomy Reference Ontology. AMIA Annual Symposium Proceedings. 2008;2008:465–9.Google Scholar
  32. Miles A, Bechhofer S: SKOS Simple Knowledge Organization System Namespace Document - HTML Variant. W3C 18 August 2009 Recommendation Edition 2009
  33. Motik B, Grau BC, Sattler U: Structured Objects in OWL: Representation and Reasoning. University of Oxford Report, UK 2007:1-44
  34. Gruninger M, Fox M: Methodology for the Design and Evaluation of Ontologies. In Proceedings of the Workshop on Basic Ontological Issues in Knowledge Sharing, IJCAI-95; Montreal, Canada.; 1995
  35. Lin EP, Bhatt S, Dogra VS. Diagnostic clues to ectopic pregnancy. Radiographics. 2008;28:1661–71.View ArticleGoogle Scholar
  36. Hripcsak G, Rothschild AS. Agreement, the f-measure, and reliability in information retrieval. J Am Med Inform Assoc. 2005;12:296–8.View ArticleGoogle Scholar
  37. Grant JM. The fetal heart rate trace is normal, isn't it? Observer agreement of categorical assessments. Lancet. 1991;337:215–8.View ArticleGoogle Scholar
  38. Berners-Lee T, Jaffe J: Semantic Web. World Wide Web Consortium, W3C website accessed 8 Dec 2015.
  39. Richardson A, Gallos I, Dobson S, Campbell BK, Coomarasamy A, Raine-Fenning N. Accuracy of first-trimester ultrasound in diagnosis of tubal ectopic pregnancy in the absence of an obvious extrauterine embryo: systematic review and meta-analysis. Ultrasound Obstet Gynecol. 2016;47:28–37.View ArticleGoogle Scholar
  40. Grenon P, Smith B, Goldberg L. Biodynamic ontology: applying BFO in the biomedical domain. Stud Health Technol Inform. 2004;102:20–38.Google Scholar
  41. Courtot M, Gibson F, Lister AL, Malone J, Schober D, Brinkman RR, Ruttenberg A. MIREOT: The minimum information to reference an external ontology term. Appl Ontol. 2011;6:23–33.Google Scholar
  42. Jurkovic D, Mavrelos D. Catch me if you scan: ultrasound diagnosis of ectopic pregnancy. Ultrasound Obstet Gynecol. 2007;30:1–7.View ArticleGoogle Scholar
  43. Kim DJ, Theoret J, Liao MM, Kendall JL. Experience with emergency ultrasound training by Canadian emergency medicine residents. West J Emerg Med. 2014;15:306–11.View ArticleGoogle Scholar
  44. Adhikari S, Blaivas M, Lyon M. Diagnosis and management of ectopic pregnancy using bedside transvaginal ultrasonography in the ED: a 2-year experience. Am J Emerg Med. 2007;25:591–6.View ArticleGoogle Scholar
  45. Kus MS, Juliano ML. Do all women with indeterminate pregnancies need a formal ultrasound before discharge from the emergency department? Mil Med. 2014;179:1263–5.View ArticleGoogle Scholar
  46. Kohler S, Schulz MH, Krawitz P, Bauer S, Dolken S, Ott CE, Mundlos C, Horn D, Mundlos S, Robinson PN. Clinical diagnostics in human genetics with semantic similarity searches in ontologies. Am J Hum Genet. 2009;85:457–64.View ArticleGoogle Scholar
  47. Budovec JJ, Lam CA, Kahn Jr CE. Informatics in radiology: radiology gamuts ontology: differential diagnosis for the Semantic Web. Radiographics. 2014;34:254–64.View ArticleGoogle Scholar


© The Author(s). 2017