Table 4 Included publications and their evaluation methodologies
Author | Year | Ref. std. | Validation | External | Generalizability a | Ref |
|---|---|---|---|---|---|---|
Afshar | 2019 | Existing EHR data | Hold-out validation (train, test, development) | No | No, validation is needed | [29] |
Alnazzawi | 2016 | Existing annotated corpus | External | ShARe/CLEF, NCBI disease, Heart failure and pulmonary embolism corpora | Yes, achieves competitive performance on other corpora | [30] |
Atutxa | 2018 | Manual retrospective review | Hold-out validation (train, test, development) | No | Yes, easily portable to other languages | [31] |
Barrett | 2013 | Manual annotations | 10-fold cross validation | Multiple datasets (different provider) | Yes, expect that it is generalizable | [32] |
Becker | 2016 | Existing annotated corpus | Not used | No | Not listed | [33] |
Becker | 2019 | Manual annotations | Hold-out validation (train, test, development) | No | Not listed | [34] |
Bejan | 2015 | Manual annotations | External | i2b2 data (2010) | Yes, good performance on the i2b2 dataset, even though not optimized on it | [35] |
Castro | 2010 | Manual annotations | Not used | No | Not listed | [36] |
Catling | 2018 | Existing annotated corpus | Hold-out validation (train, test, development) | No | Not listed | [37] |
Chapman | 2004 | Manual annotations | Not used | No | Yes, generalizable to other domains within and outside of bio surveillance | [38] |
Chen | 2016 | Manual annotations | 10-fold cross validation | No | Not listed | [39] |
Chiaramello | 2016 | Manual annotations | Not used | No | Not listed | [40] |
Chodey | 2016 | Existing annotated corpus | Hold-out validation (train, test) | No | Not listed | [41] |
Chung | 2005 | Manual annotations | Hold-out validation (train, test) | Reports from a second hospital | Not listed | [42] |
Combi | 2018 | Manual annotations | Not used | No | Not listed | [43] |
deBruijn | 2011 | Existing annotated corpus | 15-fold cross validation | No | Not listed | [44] |
Deisseroth | 2019 | Manual annotations | Hold-out validation (train, test) | Data from a second hospital | Yes, it can be immediately incorporated into clinical practice | [45] |
Demner-Fushman | 2017 | Existing annotated corpus | External | Multiple datasets | Not listed | [46] |
Divita | 2014 | Manual annotations | Not used | No | Not listed | [47] |
Duarte | 2018 | Manual annotations | Hold-out validation (train, test) | Second dataset | Not listed | [48] |
Falis | 2019 | Existing annotated corpus | Hold-out validation (train, test, development) | No | Yes, method is not specific to an ontology, and could be used for a graph of any formation | [49] |
Ferrão | 2013 | Existing EHR data | Hold-out validation (train, test) | No | Not listed | [50] |
Gerbier | 2011 | Manual annotations | Hold-out validation (train, test) | No | Yes, it could also serve other types of clinical decision support systems | [51] |
Goicoechea Salazar | 2013 | Manual annotations | Hold-out validation (train, test) | No | Not listed | [52] |
Hamid | 2013 | Manual annotations | 10-fold cross validation | No | Possible, the classifier may be applicable in academic hospital samples | [53] |
Hassanzadeh | 2016 | Existing annotated corpus | Hold-out validation (train, test) | No | Not applicable | [54] |
Helwe | 2017 | Existing annotated corpus | Hold-out validation (train, test, development) | No | Not listed | [55] |
Hersh | 2001 | Manual annotations | Hold-out validation (train, test) | No | Not listed | [56] |
Hoogendoorn | 2015 | Existing EHR data | 5-fold cross validation | No | Not listed | [57] |
Jindal | 2013 | Existing annotated corpus | Hold-out validation (train, test) | No | Yes, broad applicability | [58] |
Kang | 2009 | Manual annotations | Hold-out validation (train, test) | No | Yes, extensible to other languages | [59] |
Kersloot | 2019 | Manual annotations | Hold-out validation (development, test) | No | Possible, but external validation is needed | [60] |
König | 2019 | Existing EHR data | Not used | No | Still to be tested | [61] |
Li | 2015 | Manual annotations | 10-fold cross validation | No | Not listed | [62] |
Li | 2019 | Existing annotated corpus | Hold-out validation (train, test, development) | No | Not listed | [63] |
Lingren | 2016 | Manual annotations | Hold-out validation (train, test, development) | No | Not listed | [12] |
Liu | 2019 | Manual annotations | Not used | No (but multiple datasets / non-trained) | No, limited because of NYP/CUIMC and Mayo notes. | [64] |
Lowe | 2009 | Manual retrospective review | Hold-out validation (train, test) | No | Yes, has the potential to index other classes of clinical documents | [65] |
Luo | 2014 | Existing EHR data | 10-fold cross validation | No | No, challenging, not currently working on it | [66] |
Meystre | 2006 | Manual retrospective review | Not used | No | Not listed | [67] |
Meystre | 2010 | Existing annotated corpus | Hold-out validation (train, test) | No | Not listed | [68] |
Minard | 2011 | Existing annotated corpus | Hold-out validation (train, test, development) | No | Not listed | [69] |
Mishra | 2019 | Manual annotations | Not used | No | Not listed | [70] |
Nguyen | 2018 | Existing EHR data | Not listed | No | Not listed | [71] |
Oellrich | 2015 | Existing annotated corpus | External | Multiple datasets | Not listed | [72] |
Patrick | 2011 | Existing annotated corpus | 10-fold cross validation | No | Yes, adaptable to different requirements in clinical information extraction and classification by choosing relevant feature sets | [73] |
Pérez | 2018 | Existing annotated corpus | Hold-out validation (train, test, development) | No | Yes, extensible to different hospital-sections and hospitals | [74] |
Reátegui | 2018 | Existing annotated corpus | Not used | No | Not listed | [75] |
Roberts | 2011 | Existing annotated corpus | Hold-out validation (train, test) | No | Not listed | [76] |
Rousseau | 2019 | Manual annotations | Not used | No | Not listed | [77] |
Savova | 2010 | Manual annotations | 10-fold cross validation | No | Yes, implemented in several applications | [78] |
Shivade | 2015 | Manual annotations | Hold-out validation (train, test) | No | Not listed | [11] |
Shoenbill | 2019 | Manual annotations | Hold-out validation (train, test) | No | Yes, can allow further evaluation and improvement in care delivery models and treatment approaches to multiple chronic illnesses | [79] |
Sohn | 2014 | Manual annotations | Hold-out validation (train, test, development) | No | Yes, with adaptions: create flexible mechanism for adaptation process | [80] |
Solti | 2008 | Manual annotations | Hold-out validation (train, test) | No | Not listed | [81] |
Soriano | 2019 | Manual annotations | Not listed | No | Not listed | [82] |
Soysal | 2018 | Existing annotated corpus | Hold-out validation (train, test) | No | Yes, can be used to quickly develop customized clinical information extraction pipelines | [83] |
Spasić | 2015 | Manual annotations | Hold-out validation (train, test) | No | Not listed | [84] |
Strauss | 2013 | Manual annotations | Not used | No | Yes, can be shared between institutions and used to support clinical + epidemiological research | [85] |
Sung | 2018 | Manual annotations | Not listed | No | Not listed | [86] |
Tchechmedjiev | 2018 | Existing annotated corpus | Hold-out validation (train, test, development) | No | Yes, but not universally | [87] |
Ternois | 2018 | Existing EHR data | 5-fold cross validation + Hold-out validation (train, test) | No | Not listed | [88] |
Travers | 2004 | Manual retrospective review | Not used | No | Not listed | [89] |
Tulkens | 2019 | Existing annotated corpus | Hold-out validation (train, test, development) | No | Not listed | [90] |
Usui | 2018 | Manual annotations | Not used | No | Not listed | [91] |
Valtchinov | 2019 | Manual annotations | Not used | No | No | [92] |
Wadia | 2018 | Manual annotations | Not used | No | Not listed | [93] |
Walker | 2019 | Manual retrospective review | Hold-out validation (development, test) | No | Yes, it can be incorporated in institutional data warehouse | [94] |
Xie | 2019 | Existing annotated corpus | Hold-out validation (train, test, development) | No | Not listed | [95] |
Xu | 2011 | Manual annotations | Hold-out validation (train, test) | No | Yes, generable approach to combine information from heterogeneous data sources in EHRs | [96] |
Yadav | 2013 | Manual annotations | Not used | No | Yes, should be broadly applicate to outcomes of clinical interest | [97] |
Yao | 2019 | Existing annotated corpus | Hold-out validation (train, test) | No | Not listed | [98] |
Zeng | 2018 | Manual annotations | 5-fold cross validation + Hold-out validation (train, test) | No | Yes, potential to be replicated | [99] |
Zhang | 2013 | Existing annotated corpus | External | Two different sets with same settings | Yes, can be adapted to different semantic categories and text genres | [100] |
Zhou | 2006 | Manual annotations | 5-fold cross validation | No | Not listed | [101] |
Zhou | 2011 | Manual retrospective review | Hold-out validation (train, test) | No | Not listed | [102] |
Zhou | 2014 | Manual annotations | Not used | No | Not listed | [103] |