Skip to main content

Table 6 Evaluation methods of the included studies

From: Natural language processing algorithms for mapping clinical text fragments onto ontology concepts: a systematic review and recommendations for future studies

Description n (%) References
Evaluation: Reference standard
Manual annotations 40 (52%) [11, 12, 32, 34,35,36, 38,39,40, 42, 43, 45, 47, 48, 51,52,53, 56, 59, 60, 62, 64, 70, 77,78,79,80,81,82, 84,85,86, 91,92,93, 96, 97, 99, 101, 103]
Existing annotated corpus 24 (31%) [30, 33, 37, 41, 44, 46, 49, 55, 58, 63, 68, 69, 72,73,74,75,76, 83, 87, 90, 95, 98, 100, 104]
Existing EHR data 7 (9.1%) [29, 50, 57, 61, 66, 71, 88]
Manual retrospective review 6 (7.8%) [31, 65, 67, 89, 94, 102]
Evaluation: Validation
Hold-out validation 40 (52%) [11, 12, 29, 31, 34, 37, 41, 42, 45, 48,49,50,51,52, 55, 56, 58,59,60, 63, 65, 68, 69, 74, 76, 79,80,81, 83, 84, 87, 88, 90, 94,95,96, 98, 99, 102, 104]
Cross-validation 12 (16%) [32, 39, 44, 53, 57, 62, 66, 73, 78, 88, 99, 101]
External validation 9 (12%) [30, 32, 35, 42, 45, 46, 48, 72, 100]
Solely external validation 5 (6.5%) [30, 35, 46, 72, 100]
In addition to another type of validation 4 (5.2%) [32, 42, 45, 48]
Not performed or not listed 22 (29%) [33, 36, 38, 40, 43, 47, 61, 64, 67, 70, 71, 75, 77, 82, 85, 86, 89, 91,92,93, 97, 103]
Generalizability
Claimed 23 (30%) [30,31,32, 35, 38, 45, 49, 51, 58, 59, 65, 73, 74, 78,79,80, 83, 85, 87, 94, 96, 97, 100]
Externally validated 5 (6.5%) [30, 32, 35, 45, 100]
Comparison
Compared to other existing algorithms or models 24 (31%) [30, 35, 39, 45,46,47, 49, 58, 60, 63, 64, 72, 75, 80, 83, 87, 90, 94, 95, 98,99,100,101, 104]
Tested difference in outcomes for statistical significance 4 (5.2%) [35, 39, 60, 63]