Development and validation of the early warning system scores ontology
Journal of Biomedical Semantics volume 14, Article number: 14 (2023)
Clinical early warning scoring systems, have improved patient outcomes in a range of specializations and global contexts. These systems are used to predict patient deterioration. A multitude of patient-level physiological decompensation data has been made available through the widespread integration of early warning scoring systems within EHRs across national and international health care organizations. These data can be used to promote secondary research. The diversity of early warning scoring systems and various EHR systems is one barrier to secondary analysis of early warning score data. Given that early warning score parameters are varied, this makes it difficult to query across providers and EHR systems. Moreover, mapping and merging the parameters is challenging. We develop and validate the Early Warning System Scores Ontology (EWSSO), representing three commonly used early warning scores: the National Early Warning Score (NEWS), the six-item modified Early Warning Score (MEWS), and the quick Sequential Organ Failure Assessment (qSOFA) to overcome these problems.
We apply the Software Development Lifecycle Framework—conceived by Winston Boyce in 1970—to model the activities involved in organizing, producing, and evaluating the EWSSO. We also follow OBO Foundry Principles and the principles of best practice for domain ontology design, terms, definitions, and classifications to meet BFO requirements for ontology building.
We developed twenty-nine new classes, reused four classes and four object properties to create the EWSSO. When we queried the data our ontology-based process could differentiate between necessary and unnecessary features for score calculation 100% of the time. Further, our process applied the proper temperature conversions for the early warning score calculator 100% of the time.
Using synthetic datasets, we demonstrate the EWSSO can be used to generate and query health system data on vital signs and provide input to calculate the NEWS, six-item MEWS, and qSOFA. Future work includes extending the EWSSO by introducing additional early warning scores for adult and pediatric patient populations and creating patient profiles that contain clinical, demographic, and outcomes data regarding the patient.
Early detection and response to patient deterioration are critical to avoid adverse events, preventable morbidity, cardiac arrest, unexpected ICU admission, and mortality . Patients often display indicators of clinical deterioration up to 24 h before a significant clinical event arises needing extensive intervention . Failure to manage these deteriorations appropriately leads to negative patient outcomes . However, clinical deterioration may be masked or not obvious to the treatment team [4, 5]. Thus, the patient may deteriorate to the point of instability or even death before this deterioration is noticed and acted upon. Early Warning Scores (EWSs), also known as track and trigger systems, can assist clinicians in recognizing and responding to patient deterioration at an earlier stage of progression and in many cases improve patient care outcomes [6,7,8,9,10].
Since their introduction in the mid-1990s , different EWSs have been developed and implemented within different types of electronic health record (EHR) systems and within diverse healthcare settings. Twenty-eight different early warning system scores have been developed and validated in adult patients . Twenty-seven unique pediatric track and trigger systems have been validated in hospitalized children . The proliferation of EWSs within EHR systems across national and international health care institutions has provided a wealth of patient-level physiological decompensation clinical data that can also be used to advance secondary research, such as quality/safety protocols or population health initiatives. One roadblock to secondary analysis of EWS data is the variability of EWSs across multiple providers and multiple EHR systems. This hinders querying across providers and EHR systems, since EWS types use different parameters mapping and merging the parameters is also not always easily possible. We propose to use ontologies to overcome these problems.
In this paper we describe the development and validation of the early warning system scores ontology (EWSSO), representing three commonly used EWSs: the National Early Warning Score (NEWS), the six-item modified Early Warning Score (MEWS), and the quick Sequential Organ Failure Assessment (qSOFA) . In addition, we demonstrate how the EWSSO can be used to generate and query health system data on vital signs and calculate the NEWS, six-item MEWS, and qSOFA using synthetic data. Our goal is to establish a platform that can eventually foster semantic integration of data captured using different EWSs and generate EWS data for the secondary use of clinical data from EHR systems. The ability to semantically integrate EWS data across multiple healthcare settings with the use of ontologies will advance our goal of using real-word data from EHR systems to improve individual-level and population-level health.
Early Warning Scores, if implemented within the clinical setting, can become an integral part of the healthcare delivery process by assessing the early signs of clinical deterioration of a patient [12, 15, 16]. When EWSs are used as a track and trigger system a patient’s EWSs can either alert the healthcare team to intervene on behalf of the patient or continue to monitor the patient’s condition [12, 17, 18]. Although the validity of EWSs to predict adverse outcomes has been established across different healthcare settings and among diverse patient populations [19,20,21], these data are not being maximized by healthcare researchers and informaticists to advance secondary research of clinical data and outcomes stored within EHR systems across multiple healthcare settings.
Researchers of individual institutions may be able to extract EWS data, clean it, and append it with patient demographic and clinical characteristics using traditional data cleaning and merging techniques; but integrating data in this format, which may exist as either structured data or free text, can be inefficient and time consuming . This traditional approach to manually transform and append disparate data sources to answer a single research question can be inefficient, because the procedure is not easily reproduced by other researchers, and it does not transform heterogeneous data sources into actionable clinical information. For instance, for a researcher to have confidence in the use of a colleague's transformed dataset to extract clinical information that may be useful to answer a different scientific question, agreement with the meaning of each term used to define a particular feature must exist. Oftentimes, however, the labels or terms used by individual researchers to provide meaning to features within a dataset may differ in subtle but important ways from other researchers’ ideas. Without a standardized data dictionary that is openly accessible and includes terms agreed upon by all members of the research team and is updated overtime, use of such a data source will decrease or worse it continues to be used, but the clinical information derived is questionable. This is where ontologies can help.
Ontologies are formalized representations of the entities in a domain of discourse that are machine interpretable . They are frequently used to achieve semantic integration of biomedical data [24,25,26,27,28]. The lack of semantic integration between different EWSs implemented within different EHR systems across disparate healthcare institutions is currently hampering our ability to query EWS data to identify factors relevant to different EWSs. Our goal is to achieve semantic integration of EWS data to allow easier appending of heterogeneous patient-level and environmental-level data sets to investigate relationships that extend beyond the clinical setting. For instance, examining the correlation between aspects of social determinants, a patient’s EWSs, and negative health outcomes, such as unplanned ICU readmissions, morbidity, and mortality.
Our ontology aims to represent both the structure and planned process of the National Early Warning Score , the six-item modified Early Warning Score [30, 31], and the quick Sequential Organ Failure Assessment . The National Early Warning Score system uses inputs from seven vital function measurement datum: respiratory rate; systolic blood pressure; heart rate; Alert, Voice, Pain, Unresponsiveness (AVPU); body temperature; oxygen saturation; and supplemental oxygen use . There are several Modified Early Warning Score systems that are differentiated by the number of physiological parameters used to derive the final score. For the purposes of our study, we will focus on the six-item MEWS, which uses inputs from six vital function measurement datum, respiratory rate, systolic blood pressure, urine output, heart rate, AVPU, and body temperature . The final representation is of the qSOFA, which uses inputs from three vital function measurement datum, the Glasgow coma scale, respiratory rate, and systolic blood pressure .
Materials and methods
We followed the Software Development Life Cycle (SDLC) framework depicted in Fig. 1 to model the tasks involved in planning, creating, and testing the Early Warning Systems Score Ontology . While EWSSO is available from a public repository, we have not fully gone live with our ontology. Thus, we have not finalized the deployment, operations, and maintenance phases of the SDLC.
SDLC phase 1 and phase 2: planning and requirements
During the planning phase of the SDLC relevant information and methods to be applied were discussed to assess the feasibility of developing and validating the EWSSO. During the design phase the team agreed to apply methods described by Arp, Smith & Spear for “Building Ontologies with Basic Formal Ontology”  particularly, the principles of best practice for ontology design, terms, definitions, and classifications to meet BFO requirements for ontology building. Additionally, we followed Arp, Smith, and Spear’s guidance on using Aristotelian definitions—i.e. genus-differentia definitions—which define an A as a B with the properties C, where A is the class of things being defined (e.g., national early warning score) and B is the immediate superclass (e.g., early warning scores) and C are a set of properties that make the members of A different from the members of B that are not members of A.
As an example, two Aristotelian definitions relevant to our domain are described here.
early warning score = def. an information content entity that is about clinical deterioration.
national early warning score = def. an early warning score that is the specified output of the national early warning score summation process that has specified input respiratory rate, systolic blood pressure, heart rate, AVPU, body temperature, oxygen saturation, and supplemental oxygen use measurement data.
In the first definition, we specify A [early warning score] as a B [an information content entity] that Cs [is about clinical deterioration]. In the second definition, we specify A [national early warning score] as a B [early warning score] that Cs [is the specified output of the national early warning score summation process that has specified input respiratory rate, systolic blood pressure, heart rate, AVPU, body temperature, oxygen saturation, and supplemental oxygen use measurement data]. If we were to replace the term early warning score in the second sentence with the definition of early warning score in the first sentence the unpacked definition for national early warning score would be an information content entity that is about clinical deterioration and is the specified output of the national early warning score summation process. The result is that we have the same meaning of national early warning score as the sentence with which we began. In keeping in line with creating high-quality ontological definitions we use singular noun terms and avoid acronyms. Aristotelian definitions are helpful for ontology development and maintenance since they bind the definitions to the taxonomy underlying our ontology.
SDLC phase 3 and phase 4: designing and building
OBO foundry principles
The OBO Foundry Principles [35,36,37] guidelines are meant to serve as the norm for OBO Foundry ontologies and are used in assessing the ontologies that are submitted to the OBO Foundry for review. Because we have plans to submit the EWSSO for the Foundry's assessment, our ontology development process follows these principles. An assessment of how our ontology in the early development phase meets certain OBO Foundry principles is outlined in the results section of this paper. The OBO Foundry aims to be an orthogonal library of interactive ontologies . To prevent conflicting representation and enable optimal semantic integration among resources using OBO Foundry ontologies, a key strategy in developing an ontology for submission to the OBO Foundry is identifying potential areas of overlap with existing OBO Foundry ontologies and prevent conflicting representations . The OBO Foundry principles prescribe the reuse of relations , e.g., object properties but considering the reuse of classes is also a common and useful strategy . To address the issue of effective reuse of individual entities or sets of individual entities, OBO Foundry contributors have developed methods of importing classes, object properties, and other entities from existing ontologies , called MIREOT. There are multiple implementations of MIREOT. We use the MIREOT Protégé plugin developed by Hanna et al.  Reusing entities from preexisting ontologies using MIREOT copies over annotation values from the source ontology. Such annotation values sometimes provide cross-references to other ontologies. E.g., the EWSSO class respiratory rate measurement datum point to the class respiratory rate , which in turn points to several similar terms in medical terminologies. These cross-references are coming from an outside source and not subject to our development. Our aim is to follow the strategy to prevent duplicate representation of entities in EWSSO and OBO Foundry ontologies.
The 4 reused classes and definitions from OBO Foundry ontologies are listed in Table 1.
Reused object properties
We reused four object properties from OBO Foundry ontologies, listed in Table 2.
Every early warning score is an information content entity that is about clinical deterioration. Every early warning score is specified by property is about. The Information Artifact Ontology (IAO) defines object property is about, as “a (currently) primitive relation that relates an information artifact to an entity.” In this case, the early warning score is the information artifact that is about some entity, clinical deterioration, that is “the process of the diminishing realization of the patient's vital functions.” The Ontology for Biomedical Investigations (OBI) defines object property is specified output of as “a relation between a planned process and a continuant participating in that process. The presence of the continuant at the end of the process is explicitly specified in the objective specification which the process realizes the concretization of.” In this case, the national early warning score is the continuant at the end of the national early warning score summation process that is “…explicitly specified in the objective specification which the process realizes the concretization of.”
According to BFO, reality is made up of two types of phenomena: (1) continuants, which are things like objects, qualities, and functions that last over time; and (2) occurrents, which are the activities that continuants engage in [50,51,52]. Continuants may be independent or dependent . Entities we encounter daily (e.g., person, ball, pencil) are primary examples of independent continuants that also act as carriers of dependent continuants (e.g., a smile, the color of a ball, the ability to write) . We mention these BFO fundamental concepts of reality because we classify early warning system scores, such as the NEWS, six-item MEWS, and qSOFA as generically dependent continuants, more specifically an information content entity, that are the specified output of a planned process (occurrent) that uses attribute specific measurement inputs to calculate a patient’s early warning score, which warns about clinical deterioration.
SDLC phase 5: testing
To determine the usefulness of our ontology to generate and query heterogeneous vital sign data for early warning score calculation and secondary data analysis we performed the following steps.
Step 1: create synthetic datasets
The purpose for creating synthetic datasets was to simulate heterogeneous data sources of critical care patients whose degree of physiological decompensation is typically evaluated with an early warning system score, such as the NEWS, MEWS, and / or qSOFA. We created two different synthetic datasets. The first included vital sign measurement features specific to each of our early warning scores. The second dataset included other clinical and demographic features, unnecessary for early warning score calculator. This was done to test whether our process with ontological foundation could differentiate between features necessary and unnecessary for score calculation. Temperature measurement data were added as either degree Fahrenheit or degree Celsius. This was done so that the system would then calculate the early warning score with the appropriate temperature measurement scale. Additionally, a random number of values were left blank to represent missingness, common to all datasets. Our process is not meant to impute missing data. However, if a feature necessary to calculate an early warning score is missing the system will return the output as unavailable. Each dataset included 1,000 patient rows.
Step 2: convert synthetic dataset into triples
The synthetic datasets were converted into triples in an RDF database informed by our early warning system scores ontology. We transformed the entirety of each dataset to triples.
Step 3: visualization of RDF database and SPARQL queries
Once the heterogeneous synthetic datasets were converted into triples representing each patient and their respective vital sign measurements, the results were loaded into GraphDB. We then ran a SPARQL query to GraphDB that returned all measurement data for a specific patient (patient identifier). A visualization of the triples in the RDF database, RDF graph, SPARQL query, and the answer for the SPARQL query used to extract a person’s measurement data can be seen in Fig. 2.
Step 4: calculate the early scores for each person
To calculate the early warning scores for each person, three projects were created in Python. The intention was to define the required calculations in separate modules that would be imported and utilized by the base project. One project defined each early warning score’s requirements and the formula for the score calculation. We used the guide charts for NEWS, six-item MEWS, and qSOFA scoring system methodology provided by Aseptika Limited , the Institute for Healthcare Improvement , and Omni Calculator , respectively. Another project contained formulas for unit conversions (e.g., converting temperature to degrees Celsius when a formula requires Celsius, but the provided data is Fahrenheit). The general flow of the application was that the base Python project would query the data from the triplestore using SPARQL, representing a person with their semantically tagged measurements. The EWS-specific module would be dynamically imported by the base project and used to calculate each early warning score that meets the minimum data requirements for the score. The calculation is possible because the data representing the measurements is semantically tagged, and the formulas for each score use those tags to determine the correct parameters. In turn, the EWS module would import the unit conversion module to perform any necessary unit conversions on the data. This approach allows new early warning scores or unit conversion formulas to be added to their respective projects without interfering with the base project or other projects created to take advantage of the centralized specification.
Step 5: validate early warning system score output
Researcher CEZ randomly selected ten patients and manually calculated the early warning system scores, based on the heterogeneous datasets, and compared to system output.
OBO foundry principles
The list below indicates how the early-stage development of the EWSSO fulfills several of the OBO Foundry Principles.
P1) Open. Our ontology is openly available to all users under the Creative Commons CC-BY 3.0 license. This means it may be shared and adopted if appropriately credited and changes, if made, are indicated.
P2) Common Format. Our ontology is made available in the OWL Web Ontology Language and the Resource Description Framework (RDF).
P5) Scope. The domain of the ontology is early warning system scores for clinical deterioration in inpatient populations. We start with the simplest and most easily defined features of our relevant domain and will work our way outward to more complex features in future updates of our ontology.
P6) Textual Definitions. All newly created classes have a textual definition (see Table 3).
P7) Relations. Relations reused from the Relations Ontology (RO), Basic Formal Ontology (BFO), Ontology for Biomedical Investigations (OBI), and Information Artifact Ontology (IAO).
Our long-term goal is to submit the EWSSO to the OBO Foundry. Hence, the GitHub repository is created following OBO Foundry practices and rules. The repository is modeled to be similar to GitHub repositories, such as OBI [https://github.com/obi-ontology/obi] or OBIB [https://github.com/biobanking/biobanking] among others.
EWSSO is freely available on the GitHub https://github.com/zcilia/ews-ontology.
The data for our experiment can be found here: https://github.com/EWSSO/documentation
Ontosheep (application full instructions): https://github.com/jmwhorton/ontosheep/tree/main
Ontosheep (container instructions): https://github.com/jmwhorton/ontosheep/tree/main/test-deploy
Ontosheep modules (These are the two 'helper' modules for Ontosheep. Modules in separate projects for reusabiity.): https://github.com/jmwhorton/ontosheep-ews and https://github.com/jmwhorton/ontosheep-conversion
Newly created classes
Table 3 lists the 29 new classes created and their respective definitions. We also include references reviewed that inspired our new definitions, or the Internationalized Resource Identifier (IRI) of the OBO Foundry ontology classes that we based our new definitions on.
Representation of early warning system score ontology
Ontology performance evaluation
Our evaluation of EWSSO was to generate and query health systems data and correctly retrieve the data elements required to calculate the NEWS, six-item MEWS, and qSOFA and no other data elements. To test performance, we created two synthetic datasets with 1,000 patient rows each. The first synthetic dataset included vital sign measurement features specific to each of our early warning scores. The second synthetic dataset included other clinical and demographic features, unnecessary for early warning score calculator. Each time we queried the data, our ontology-based process could differentiate between features necessary and unnecessary for score calculation.
One of the core terms for early warning scores is clinical deterioration. We checked multiple ontology lookup resources, e.g., OntoBee  and the NCBO Bioportal  for a term to re-use. The only term we could find was from Medical Subject Headings (MeSH) resource and defines clinical deterioration as “A critical disease progression, often measured by a set of clinical parameters, which activates HOSPITAL RAPID RESPONSE TEAM” . This definition is not optimal, since it defines clinical deterioration by the activation of a hospital rapid response team. The authors hold that clinical deterioration may exist without a hospital rapid response team present and activated. A recent concept analysis of clinical deterioration did not result in a definition referring to the activation of a hospital rapid response team or any other healthcare procedure . We propose to define clinical deterioration as the process of the diminishing realization of the patient’s vital functions.
While the current project is of limited scope, as only NEWS, MEWS, and qSOFA are considered it serves as the basis for future work with all early warning scoring systems. As the project is consolidated, we are going to seek to integrate additional early warning scores. Although the EWSS ontology does not perform mathematical calculations, it is expected to help users query relevant ontological relationships and measurement data. Using Microsoft Excel, SAS, or other computer programming languages researchers can use the queried data to calculate the early warning scores for secondary analyses of clinical data.
It is our plan to create a pipeline that allows clinical service providers and clinical researchers to perform data acquisition and EWS calculation without the need to run SPARQL queries or write Python commands. During the development of this pipeline, we will recruit a small number of independent users to test it and provide feedback on its usability. During these trials EWSSO will begin to fulfill the independent user requirement of the OBO Foundry. Once EWSSO fulfills all criteria for applying for an OBO Foundry namespace we will apply for a namespace—preferably “EWSSO”—assigned by the OBO Foundry.
The objective of this paper was to describe the development and validation of the early warning system scores ontology using the principles of best practice for domain ontology design, terms, definitions, and classification as described by Arp, Smith & Spear Building Ontologies with Basic Formal Ontology  and aiming to follow OBO Foundry principles [35,36,37]. We have developed the EWSS Ontology covering three commonly used early warning scores in clinical practice: NEWS, six-item MEWS, and qSOFA. We demonstrated that EWSSO provides the functionality to facilitate search and extraction of data necessary to calculate the scores.
Further work includes extending the EWSS Ontology by introducing additional early warning scores for adult and pediatric patient populations and creating patient profiles which contain clinical, demographic, and outcomes data regarding the patient. The early warning system scores ontology can be implemented and personalized for a particular healthcare system by adding classes for organizational characteristics including clinical staff, non-clinical staff, and management staff. Users of the ontology can develop queries based on the semantically integrated data created, thereby reducing the inefficiency involved with compiling voluminous datasets of heterogeneous EHR data for research or quality improvement initiatives.
Availability of data and materials
EWSSO is freely available on the GitHub https://github.com/zcilia/ews-ontology.
The data for our experiment can be found here:https://github.com/EWSSO/documentation.
• Ontosheep (application full instructions):https://github.com/jmwhorton/ontosheep/tree/main.
• Ontosheep (container instructions):https://github.com/jmwhorton/ontosheep/tree/main/test-deploy.
• Ontosheep modules (These are the two 'helper' modules for Ontosheep. Modules in separate projects for reusabiity.):https://github.com/jmwhorton/ontosheep-ews and https://github.com/jmwhorton/ontosheep-conversion.
Basic Formal Ontology
Early Warning Systems Score Ontology
Information Artifact Ontology
Modified Early Warning Score
National Early Warning Score
Ontology for Biomedical Investigations
Quick Sepsis Related Organ Failure Assessment
Maharaj R, Raffaele I, Wendon J. Rapid response systems: a systematic review and meta-analysis. Crit Care. 2015;19(1):254. https://doi.org/10.1186/s13054-015-0973-y.
McGaughey J, Alderdice F, Fowler R, Kapila A, Mayhew A, Moutray M. “Outreach and Early Warning Systems (EWS) for the prevention of intensive care admission and death of critically ill adult patients on general hospital wards,” Cochrane Database Syst Rev 2007(3);CD005529. https://doi.org/10.1002/14651858.CD005529.pub2.
Sun L, Joshi M, Khan SN, Ashrafian H, Darzi A. Clinical impact of multi-parameter continuous non-invasive monitoring in hospital wards: a systematic review and meta-analysis. J R Soc Med. 2020;113(6):217–24. https://doi.org/10.1177/0141076820925436.
Smith AF, Wood J. Can some in-hospital cardio-respiratory arrests be prevented? A prospective survey. Resuscitation. 1998;37(3):133–7. https://doi.org/10.1016/S0300-9572(98)00056-2.
Buist MD, Burton PR, Bernard SA, Waxman BP, Anderson J. Recognising clinical instability in hospital patients before cardiac arrest or unplanned admission to intensive care: a pilot study in a tertiary-care hospital. Med J Aust. 1999;171(1):22–5. https://doi.org/10.5694/j.1326-5377.1999.tb123492.x.
Credland N, Dyson J, Johnson MJ. Do early warning track and trigger tools improve patient outcomes? A systematic synthesis without meta-analysis. J Adv Nurs. 2021;77(2):622–34. https://doi.org/10.1111/jan.14619.
Penney SW, et al. A quality improvement initiative to reduce unnecessary rapid responses using early warning scores. Pediatrics. 2021;147(3):e20191947. https://doi.org/10.1542/peds.2019-1947.
Wu C-L, et al. Implementation of an electronic national early warning system to decrease clinical deterioration in hospitalized patients at a tertiary medical center. Int J Environ Res Public Health. 2021;18(9):4550. https://doi.org/10.3390/ijerph18094550.
Mathukia C, Fan W, Vadyak K, Biege C, Krishnamurthy M. Modified early warning system improves patient safety and clinical outcomes in an academic community hospital. J Commun Hosp Internal Med Perspect. 2015;5:26716. https://doi.org/10.3402/jchimp.v5.26716.
Nishijima I, et al. Use of a modified early warning score system to reduce the rate of in-hospital cardiac arrest. J Intensive Care. 2016;4:12. https://doi.org/10.1186/s40560-016-0134-7.
Lee A, Bishop G, Hillman KM, Daffurn K. The medical emergency team. Anaesth Intensive Care. 1995;23(2):183–6. https://doi.org/10.1177/0310057X9502300210.
Wuytack F, et al. The effectiveness of physiologically based early warning or track and trigger systems after triage in adult patients presenting to emergency departments: a systematic review. BMC Emerg Med. 2017;17:38. https://doi.org/10.1186/s12873-017-0148-z.
Trubey R, et al. Validity and effectiveness of paediatric early warning systems and track and trigger tools for identifying and reducing clinical deterioration in hospitalised children: a systematic review. BMJ Open. 2019;9(5):e022105.https://doi.org/10.1136/bmjopen-2018-022105.
Liu VX, et al. Comparison of early warning scoring systems for hospitalized patients with and without infection at risk for in-hospital mortality and transfer to the intensive care unit. JAMA Netw Open. 2020;3(5):e205191. https://doi.org/10.1001/jamanetworkopen.2020.5191.
Burgos-Esteban A, et al. Effectiveness of early warning scores for early severity assessment in outpatient emergency care: a systematic review. Front Public Health. 2022;10:894906. https://doi.org/10.3389/fpubh.2022.894906.
Cooksley T, Kitlowski E, Haji-Michael P. Effectiveness of modified early warning score in predicting outcomes in oncology patients. QJM. 2012;105(11):1083–8. https://doi.org/10.1093/qjmed/hcs138.
M. E. B. Smith et al., Early Warning System Scores: A Systematic Review. in VA Evidence-based Synthesis Program Reports. Washington (DC): Department of Veterans Affairs, 2014. Available: http://www.ncbi.nlm.nih.gov/books/NBK259026/.Accessed 11 Apr 2022
F. Creed, J. Dawson, and K. Looker, “Assessment tools and track-and-trigger systems,” in Care of the Acutely Ill Adult: An essential guide for nurses, F. Creed and C. Spiers, Eds., Oxford University Press, 2010 0. https://doi.org/10.1093/med/9780199564385.003.11.
Fang AHS, Lim WT, Balakrishnan T. Early warning score validation methodologies and performance metrics: a systematic review. BMC Med Inform Decis Mak. 2020;20:111. https://doi.org/10.1186/s12911-020-01144-8.
Gerry S, Birks J, Bonnici T, Watkinson PJ, Kirtley S, Collins GS. Early warning scores for detecting deterioration in adult hospital patients: a systematic review protocol. BMJ Open. 2017;7(12):e019268. https://doi.org/10.1136/bmjopen-2017-019268.
Alhmoud B, Bonnici T, Patel R, Melley D, Williams B, Banerjee A. Performance of universal early warning scores in different patient subgroups and clinical settings: a systematic review. BMJ Open. 2021;11(4):e045849. https://doi.org/10.1136/bmjopen-2020-045849.
Ridzuan F, Zainon WM. A review on data cleansing methods for big data. Proc Comp Sci. 2019;161:731–8. https://doi.org/10.1016/j.procs.2019.11.177.
Gruber TR. Toward principles for the design of ontologies used for knowledge sharing? Int J Hum Comput Stud. 1995;43(5):907–28. https://doi.org/10.1006/ijhc.1995.1081.
Livingston KM, Bada M, Baumgartner WA, Hunter LE. KaBOB: ontology-based semantic integration of biomedical databases. BMC Bioinform. 2015;16(1):126. https://doi.org/10.1186/s12859-015-0559-3.
Bona J, et al. Semantic integration of multi-modal data and derived neuroimaging results using the Platform for Imaging in Precision Medicine (PRISM) in the Arkansas Imaging Enterprise System (ARIES). Front Artif Intell. 2021;4:649970. https://doi.org/10.3389/frai.2021.649970.
Bona JP, Prior FW, Zozus MN, Brochhausen M. Enhancing clinical data and clinical research data with biomedical ontologies - insights from the knowledge representation perspective. Yearb Med Inform. 2019;28(1):140–51. https://doi.org/10.1055/s-0039-1677912.
Zaman S, Sarntivijai S, Abernethy DR. Use of biomedical ontologies for integration of biological knowledge for learning and prediction of adverse drug reactions. Gene Regul Syst Bio. 2017;11:1177625017696075. https://doi.org/10.1177/1177625017696075.
O. Bodenreider, “Biomedical Ontologies in Action: Role in Knowledge Management, Data Integration and Decision Support,” Yearb Med Inform 2008;67–79. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2592252/. Accessed 15 Jul 2023
Royal College of Physicians of London. National Early Warning Score (NEWS): standardising the assessment of acute-illness severity in the NHS. London: Royal College of Physicians; 2012.
Morgan R, Williams F, Wright M. An early warning scoring system for detecting developing critical illness. Clin Intensive Care. 1997;8(2):100.
Gardner-Thorpe J, Love N, Wrightson J, Walsh S, Keeling N. The value of Modified Early Warning Score (MEWS) in surgical in-patients: a prospective observational study. Ann R Coll Surg Engl. 2006;88(6):571–5. https://doi.org/10.1308/003588406X130615.
Wagner T, et al. qSOFA score is useful to assess disease severity in patients with heart failure in the setting of a Heart Failure Unit (HFU). Front Cardiovasc Med. 2020;7:574768. https://doi.org/10.3389/fcvm.2020.574768.
Laveena Ramchandani, “The World of Software Development Life Cycle,” TestProject by Tricentis. https://blog.testproject.io/2021/03/08/the-world-of-software-development-life-cycle/ (Accessed 16 Mar 2023).
Arp R, Smith B, Spear AD. Building Ontologies With Basic Formal Ontology. The MIT Press. 2015. https://doi.org/10.7551/mitpress/9780262527811.001.0001.
“OBO Foundry: Priciples Overview.” https://obofoundry.org/principles/fp-000-summary.html(Accessed 16 Feb 2023).
Jackson R, et al. OBO foundry in 2021: operationalizing open data principles to evaluate ontologies. Database (Oxford). 2021;2021:069. https://doi.org/10.1093/database/baab069.
Smith B, et al. The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration. Nat Biotechnol. 2007;25(11):1251–5. https://doi.org/10.1038/nbt1346.
Courtot M, et al. “MIREOT: the minimum information to reference an external ontology term,” Nat Prec 2009;1–1. https://doi.org/10.1038/npre.2009.3576.1.
Hanna J, Chen C, Crow WA, Hall R, Liu J, Pendurthi T, Schmidt T, Jennings SF, Brochhausen M, Hogan W., “Simplifying MIREOT: A MIREOT Protégé Plugin. Presented at the International Semantic Web Conference, Boston, Massachusetts. NIHMSID: 446691.,”
“respiratory rate.” https://www.ontobee.org/ontology/HTN?iri=http://purl.obolibrary.org/obo/VSO_0000034 (Accessed 10 May 2022).
“process.” http://purl.obolibrary.org/obo/BFO_0000015 (Accessed 16 Jul 2023).
“planned process.” http://purl.obolibrary.org/obo/OBI_0000011 (Accessed 16 Jul 2023).
“information content entity.” http://purl.obolibrary.org/obo/IAO_0000027 (Accessed 16 Jul 2023).
“measurement datum.” http://purl.obolibrary.org/obo/IAO_0000109 (Accessed 16 Jul 2023).
“‘is_about.’” http://purl.obolibrary.org/obo/IAO_0000136 (Accessed 10 May 2022).
“‘is_specified_output_of.’” http://purl.obolibrary.org/obo/OBI_0000312 (Accessed 10 May 2022).
“‘has_specified_input.’” http://purl.obolibrary.org/obo/OBI_0000293 (Accessed 10 May 2022).
“is_specified_input_of.” http://purl.obolibrary.org/obo/OBI_0000295 (Accessed 16 Jul 2023).
“‘has_specified_output.’” http://purl.obolibrary.org/obo/OBI_0000299 (Accessed 10 May 2022).
Grenon P, Smith B. SNAP and SPAN: towards dynamic spatial ontology. Spat Cogn Comput. 2004;4(1):69–104. https://doi.org/10.1207/s15427633scc0401_5.
Smith B, Grenon P. The Cornucopia of formal-ontological relations. Dialectica. 2004;58(3):279–96. https://doi.org/10.1111/j.1746-8361.2004.tb00305.x.
Arp R, Smith B. “Realizable entities in basic formal ontology” 2008.
Musen MA. The protégé project: a look back and a look forward. AI Matters. 2015;1(4):4–12. https://doi.org/10.1145/2757001.2757003.
Dinesh, “Answer to ‘How to create a list of numbers with given range, quartiles and median?,’” Stack Overflow. https://stackoverflow.com/a/26478053 (Accessed 17 Mar 2023).
Nissen SK, et al. The impact of age on predictive performance of national early warning score at arrival to emergency departments: development and external validation. Ann Emerg Med. 2022;79(4):354–63. https://doi.org/10.1016/j.annemergmed.2021.09.434.
D. Romanelli and M. W. Farrell, “AVPU Score,” in StatPearls, Treasure Island (FL): StatPearls Publishing, 2022. Available: http://www.ncbi.nlm.nih.gov/books/NBK538431/. Accessed 16 Feb 2023.
“National Early Warning Score (NEWS).” https://www.activ8rlives.com/support/data-collected/cardiovascular-and-respiratory/national-early-warning-score-news (Accessed 16 Feb 2023).
“Early Warning Systems: Scorecards That Save Lives | IHI - Institute for Healthcare Improvement.” https://www.ihi.org:443/resources/Pages/ImprovementStories/EarlyWarningSystemsScorecardsThatSaveLives.aspx (Accessed 16 Feb 2023).
Pawlik, K, “qSOFA Score Calculator | Formula | Definition.” https://www.omnicalculator.com/health/q-sofa (Accessed 16 Feb 2023).
Padilla RM, Mayo AM. Clinical deterioration: a concept analysis. J Clin Nurs. 2018;27(7–8):1360–8. https://doi.org/10.1111/jocn.14238.
“Alert verbal painful unresponsive scale clinical classification.” http://purl.obolibrary.org/obo/NCIT_C161656 (Accessed 10 May 2022).
“Glasgow coma scale-NINDS version clinical classification.” http://purl.obolibrary.org/obo/NCIT_C124710 (Accessed 10 May 2022).
“Oxygen Saturation Measurement.” http://purl.obolibrary.org/obo/NCIT_C60832 (Accessed 10 May 2022).
“body temperature.” https://www.ontobee.org/ontology/HTN?iri=http://purl.obolibrary.org/obo/VSO_0000007 (Accessed 10 May 2022).
T. D. Homan, S. Bordes, and E. Cichowski, “Physiology, Pulse Pressure,” in StatPearls, Treasure Island (FL): StatPearls Publishing, 2022. Available: http://www.ncbi.nlm.nih.gov/books/NBK482408/. Accessed 10 May 2022
“Heart Rate.” http://purl.obolibrary.org/obo/NCIT_C49677 (Accessed 10 May 2022).
Z. Xiang, C. Mungall, A. Ruttenberg, and Y. He, “Ontobee: A Linked Data Server and Browser for Ontology Terms,” presented at the International Conference on Biomedical Ontology, 2011. Available: https://www.semanticscholar.org/paper/Ontobee%3A-A-Linked-Data-Server-and-Browser-for-Terms-Xiang-Mungall/4b32148dc40590475444881ae92b379ba71f857b. Accessed 20 Apr 2023.
Noy NF, et al. BioPortal: ontologies and integrated data resources at the click of a mouse. Nucleic Acids Res. 2009;37(Web Server issue):170–3. https://doi.org/10.1093/nar/gkp440.
“Medical Subject Headings - Clinical Deterioration - Classes | NCBO BioPortal.” https://bioportal.bioontology.org/ontologies/MESH?p=classes&conceptid=http%3A%2F%2Fpurl.bioontology.org%2Fontology%2FMESH%2FD000075902 (Accessed 26 Apr 2023).
Ethics approval and consent to participate
Consent for publication
The authors declare no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Zayas, C.E., Whorton, J.M., Sexton, K.W. et al. Development and validation of the early warning system scores ontology. J Biomed Semant 14, 14 (2023). https://doi.org/10.1186/s13326-023-00296-6