Natural language processing algorithms for mapping clinical text fragments onto ontology concepts: a systematic review and recommendations for future studies

Table 6 Evaluation methods of the included studies

Description	n (%)	References
Evaluation: Reference standard
Manual annotations	40 (52%)	[11, 12, 32, 34,35,36, 38,39,40, 42, 43, 45, 47, 48, 51,52,53, 56, 59, 60, 62, 64, 70, 77,78,79,80,81,82, 84,85,86, 91,92,93, 96, 97, 99, 101, 103]
Existing annotated corpus	24 (31%)	[30, 33, 37, 41, 44, 46, 49, 55, 58, 63, 68, 69, 72,73,74,75,76, 83, 87, 90, 95, 98, 100, 104]
Existing EHR data	7 (9.1%)	[29, 50, 57, 61, 66, 71, 88]
Manual retrospective review	6 (7.8%)	[31, 65, 67, 89, 94, 102]
Evaluation: Validation
Hold-out validation	40 (52%)	[11, 12, 29, 31, 34, 37, 41, 42, 45, 48,49,50,51,52, 55, 56, 58,59,60, 63, 65, 68, 69, 74, 76, 79,80,81, 83, 84, 87, 88, 90, 94,95,96, 98, 99, 102, 104]
Cross-validation	12 (16%)	[32, 39, 44, 53, 57, 62, 66, 73, 78, 88, 99, 101]
External validation	9 (12%)	[30, 32, 35, 42, 45, 46, 48, 72, 100]
Solely external validation	5 (6.5%)	[30, 35, 46, 72, 100]
In addition to another type of validation	4 (5.2%)	[32, 42, 45, 48]
Not performed or not listed	22 (29%)	[33, 36, 38, 40, 43, 47, 61, 64, 67, 70, 71, 75, 77, 82, 85, 86, 89, 91,92,93, 97, 103]
Generalizability
Claimed	23 (30%)	[30,31,32, 35, 38, 45, 49, 51, 58, 59, 65, 73, 74, 78,79,80, 83, 85, 87, 94, 96, 97, 100]
Externally validated	5 (6.5%)	[30, 32, 35, 45, 100]
Comparison
Compared to other existing algorithms or models	24 (31%)	[30, 35, 39, 45,46,47, 49, 58, 60, 63, 64, 72, 75, 80, 83, 87, 90, 94, 95, 98,99,100,101, 104]
Tested difference in outcomes for statistical significance	4 (5.2%)	[35, 39, 60, 63]

ISSN: 2041-1480