Table 7 Performance measures used in the included studies
Description | Formula | n (%) | References |
|---|---|---|---|
Confusion Matrix | Lists the True Positives (TP), True Negatives (TN), False Positives (FP), False Negatives (FN), and the Total (n) amount in a 2 × 2 contingency Table. TP: Text annotated with ontology concept when ontology concept is present in reference standard TN: Text not annotated with ontology concept when ontology concept is absent in reference standard FP: Text annotated with ontology concept when ontology concept is absent in reference standard FN: Text not annotated with ontology concept when ontology concept is present in reference standard | 12 (16%) | |
Performance measures | |||
Recall | \( \frac{TP}{FN+ TP} \) | 68 (88%) | [11, 12, 29,30,31, 33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53, 56,57,58, 60,61,62,63,64, 66,67,68,69,70,71,72,73, 75,76,77,78,79,80,81,82,83,84,85,86,87,88, 90,91,92,93,94, 96, 99,100,101,102,103,104] |
Precision | \( \frac{TP}{FP+ TP} \) | 66 (86%) | [11, 12, 29,30,31, 33,34,35,36, 38,39,40,41,42,43,44,45,46,47,48,49,50,51, 53, 56,57,58, 60,61,62,63,64,65,66,67,68,69,70,71,72,73, 75,76,77,78,79,80,81,82,83,84,85,86,87,88, 90, 91, 93, 94, 96, 99,100,101,102,103,104] |
F-score | \( 2\bullet \frac{Precision\bullet Recall}{Precision+ Recall} \) | 57 (74%) | [11, 12, 30, 31, 33,34,35,36, 39,40,41, 44, 46,47,48,49,50, 52, 53, 55, 57,58,59,60,61,62,63, 66,67,68,69,70,71,72,73, 75,76,77,78,79,80, 82,83,84, 86,87,88, 90, 91, 95, 96, 98,99,100, 102,103,104] |
Accuracy | \( \frac{TP+ TN}{n} \) | 11 (14%) | |
Specificity | \( \frac{TN}{FP+ TN} \) | 6 (7.8%) | |
AUC | Not applicable | 5 (6.5%) | |
Kappa | \( \frac{p_o-\kern0.5em {p}_e}{1-{p}_e}=1-\frac{1-{p}_o}{1-{p}_e} \) | 3 (3.9%) | |
Processing time | Not applicable | 3 (3.9%) | |
Negative Predictive Value | \( \frac{TN}{FN+ TN} \) | 3 (3.9%) | |
False Positive Rate | \( \frac{FP}{FP+ TN} \) | 1 (1.3%) | [34] |
False Negative Rate | \( \frac{FN}{TP+ FN} \) | 1 (1.3%) | [34] |
Information entropy | \( -{\sum}_{i=1}^n{P}_i\ \mathit{\log}\left({P}_i\right) \) | 1 (1.3%) | [64] |
Mean Reciprocal Rank | \( \frac{1}{Q}{\sum}_{i=1}^Q\frac{1}{{\mathit{\operatorname{rank}}}_i} \) | 1 (1.3%) | [74] |
Initial annotator agreement | Not applicable | 1 (1.3%) | [79] |
Match/no match (%) | Not applicable | 1 (1.3%) | [89] |
Overgeneration | \( \frac{FP}{TP+ FP} \) | 1 (1.3%) | [93] |
Undergeneration | \( \frac{FN}{TP+ FN} \) | 1 (1.3%) | [68] |
Error | \( \frac{FN+ FP}{TP+ FN+ FP} \) | 1 (1.3%) | [68] |
Fallout | \( \frac{FP}{TN+ FP} \) | 1 (1.3%) | [68] |
Mean Standard Error | \( \frac{1}{n}{\sum}_{i=1}^n{\left({Y}_i-{\hat{Y}}_i\right)}^2 \) | 1 (1.3%) | [57] |