# Table 7 Performance measures used in the included studies

Description Formula n (%) References
Confusion Matrix Lists the True Positives (TP), True Negatives (TN), False Positives (FP), False Negatives (FN), and the Total (n) amount in a 2 × 2 contingency Table.
TP: Text annotated with ontology concept when ontology concept is present in reference standard
TN: Text not annotated with ontology concept when ontology concept is absent in reference standard
FP: Text annotated with ontology concept when ontology concept is absent in reference standard
FN: Text not annotated with ontology concept when ontology concept is present in reference standard
12 (16%) [34, 44, 47, 51, 56, 58, 60, 61, 84, 87, 91, 93]
Performance measures
Recall $$\frac{TP}{FN+ TP}$$ 68 (88%) [11, 12, 29,30,31, 33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53, 56,57,58, 60,61,62,63,64, 66,67,68,69,70,71,72,73, 75,76,77,78,79,80,81,82,83,84,85,86,87,88, 90,91,92,93,94, 96, 99,100,101,102,103,104]
Precision $$\frac{TP}{FP+ TP}$$ 66 (86%) [11, 12, 29,30,31, 33,34,35,36, 38,39,40,41,42,43,44,45,46,47,48,49,50,51, 53, 56,57,58, 60,61,62,63,64,65,66,67,68,69,70,71,72,73, 75,76,77,78,79,80,81,82,83,84,85,86,87,88, 90, 91, 93, 94, 96, 99,100,101,102,103,104]
F-score $$2\bullet \frac{Precision\bullet Recall}{Precision+ Recall}$$ 57 (74%) [11, 12, 30, 31, 33,34,35,36, 39,40,41, 44, 46,47,48,49,50, 52, 53, 55, 57,58,59,60,61,62,63, 66,67,68,69,70,71,72,73, 75,76,77,78,79,80, 82,83,84, 86,87,88, 90, 91, 95, 96, 98,99,100, 102,103,104]
Accuracy $$\frac{TP+ TN}{n}$$ 11 (14%) [30, 32, 34, 41, 48, 52, 67, 74, 78, 92, 96]
Specificity $$\frac{TN}{FP+ TN}$$ 6 (7.8%) [29, 34, 85, 92, 93, 96]
AUC Not applicable 5 (6.5%) [29, 39, 57, 95, 99]
Kappa $$\frac{p_o-\kern0.5em {p}_e}{1-{p}_e}=1-\frac{1-{p}_o}{1-{p}_e}$$ 3 (3.9%) [85, 89, 97]
Processing time Not applicable 3 (3.9%) [32, 47, 83]
Negative Predictive Value $$\frac{TN}{FN+ TN}$$ 3 (3.9%) [29, 85, 93]
False Positive Rate $$\frac{FP}{FP+ TN}$$ 1 (1.3%) [34]
False Negative Rate $$\frac{FN}{TP+ FN}$$ 1 (1.3%) [34]
Information entropy $$-{\sum}_{i=1}^n{P}_i\ \mathit{\log}\left({P}_i\right)$$ 1 (1.3%) [64]
Mean Reciprocal Rank $$\frac{1}{Q}{\sum}_{i=1}^Q\frac{1}{{\mathit{\operatorname{rank}}}_i}$$ 1 (1.3%) [74]
Initial annotator agreement Not applicable 1 (1.3%) [79]
Match/no match (%) Not applicable 1 (1.3%) [89]
Overgeneration $$\frac{FP}{TP+ FP}$$ 1 (1.3%) [93]
Undergeneration $$\frac{FN}{TP+ FN}$$ 1 (1.3%) [68]
Error $$\frac{FN+ FP}{TP+ FN+ FP}$$ 1 (1.3%) [68]
Fallout $$\frac{FP}{TN+ FP}$$ 1 (1.3%) [68]
Mean Standard Error $$\frac{1}{n}{\sum}_{i=1}^n{\left({Y}_i-{\hat{Y}}_i\right)}^2$$ 1 (1.3%) [57]