Table 6 Evaluation results for each tag and in total, for different methods (rule, CRF, LSTM) and different evaluation datasets (MedNLP, dummy EHR, and pathology reports). M, d, and P respectively denote training data of MedNLP, dummy EHR, and Pathology reports; M + d denotes that training data consist of MedNLP+dummy EHR, all stands for all of these three datasets; other machine learning methods use the target evaluation dataset as its training data. In each cell, F1-score, precision, and recall are shown (in values multiplied by 100). The best scores for each tag type for each evaluation metric are presented in bold typeface. All evaluations were done by four-fold cross validations
From: De-identifying free text of Japanese electronic health records
Evaluation Results on MedNLP dataset | |||||||||||||
tag type | #of tags | scores | Rule | CRF | CRF d | CRF P | CRF M + d | CRF all | LSTM | LSTM d | LSTM P | LSTM M + d | LSTM all |
total | 490 | F1 | 84.23 | 82.62 | 43.85 | 0.71 | 26.40 | 67.34 | 83.07 | 41.26 | 0.43 | 67.35 | 57.03 |
prec | 78.90 | 85.63 | 46.20 | 2.50 | 21.51 | 66.54 | 81.33 | 41.07 | 0.48 | 66.98 | 57.94 | ||
recall | 90.42 | 79.95 | 42.33 | 0.41 | 59.76 | 68.38 | 86.12 | 41.57 | 0.38 | 68.17 | 56.34 | ||
age | 56 | F1 | 93.43 | 71.12 | 30.00 | 0.00 | 32.55 | 53.04 | 95.83 | 71.11 | 0.00 | 84.72 | 87.50 |
prec | 96.00 | 78.24 | 37.50 | 0.00 | 26.93 | 56.85 | 95.83 | 71.11 | 0.00 | 84.72 | 87.50 | ||
recall | 91.16 | 65.47 | 28.13 | 0.00 | 46.05 | 50.00 | 95.83 | 71.11 | 0.00 | 84.72 | 87.50 | ||
hospital | 75 | F1 | 84.73 | 87.09 | 43.25 | 0.00 | 26.02 | 70.04 | 66.67 | 13.33 | 13.89 | 66.67 | 41.67 |
prec | 80.75 | 93.52 | 66.67 | 0.00 | 20.55 | 91.67 | 75.00 | 11.11 | 10.67 | 70.83 | 45.83 | ||
recall | 89.90 | 81.71 | 27.50 | 0.00 | 53.06 | 60.42 | 62.50 | 16.67 | 20.00 | 63.89 | 38.89 | ||
person | 0 | Â | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
sex | 4 | F1 | 50.00 | 16.67 | 16.67 | 0.00 | 14.65 | 25.00 | 0.00 | 20.00 | 0.00 | 25.00 | 25.00 |
prec | 50.00 | 25.00 | 12.50 | 0.00 | 8.68 | 25.00 | 0.00 | 20.00 | 0.00 | 25.00 | 25.00 | ||
recall | 50.00 | 12.50 | 25.00 | 0.00 | 50.00 | 25.00 | 0.00 | 20.00 | 0.00 | 25.00 | 25.00 | ||
time | 355 | F1 | 50.00 | 16.67 | 47.43 | 0.98 | 14.65 | 70.57 | 96.14 | 67.22 | 42.98 | 89.78 | 82.67 |
prec | 50.00 | 25.00 | 45.16 | 2.50 | 8.68 | 65.46 | 95.00 | 66.26 | 39.46 | 88.68 | 81.53 | ||
recall | 50.00 | 12.50 | 50.19 | 0.61 | 50.00 | 76.50 | 97.41 | 68.30 | 47.94 | 91.00 | 82.67 | ||
Evaluation Results on Pathology Report dataset | |||||||||||||
tag type | #of tags | scores | Rule | CRF | CRF M | CRF d | CRF M + d | CRF all | LSTM | LSTM M | LSTM d | LSTM M + d | LSTM all |
all | 71 | F1 | 13.97 | 74.26 | 0.00 | 0.62 | 1.45 | 57.63 | 81.67 | 0.00 | 0.00 | 1.45 | 81.25 |
prec | 8.65 | 86.72 | 0.00 | 1.47 | 10.00 | 64.98 | 86.88 | 0.00 | 0.00 | 10.00 | 82.48 | ||
recall | 43.33 | 65.16 | 0.00 | 0.39 | 0.78 | 54.06 | 78.84 | 0.00 | 0.00 | 0.78 | 80.15 | ||
age | 0 | Â | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
hospital | 31 | F1 | 31.19 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 25.00 | 0.00 | 13.33 | 0.00 | 58.33 |
prec | 26.47 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 25.00 | 0.00 | 13.33 | 0.00 | 58.33 | ||
recall | 41.28 | 0.00 | 0.00 | 0.00 | 0.000 | 0.00 | 25.00 | 0.00 | 13.33 | 0.00 | 58.33 | ||
person | 224 | F1 | 0.00 | 91.08 | 0.00 | 0.00 | 6.25 | 71.31 | 95.19 | 0.00 | 0.00 | 0.00 | 95.83 |
prec | 0.00 | 95.83 | 0.00 | 0.00 | 10.00 | 74.79 | 95.19 | 0.00 | 0.00 | 0.00 | 95.83 | ||
recall | 0.00 | 87.21 | 0.00 | 0.00 | 4.55 | 69.63 | 95.19 | 0.00 | 0.00 | 0.00 | 95.83 | ||
sex | 0 | Â | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
time | 40 | F1 | 9.25 | 10.57 | 0.00 | 2.00 | 0.00 | 18.82 | 25.00 | 3.81 | 0.00 | 6.25 | 19.44 |
prec | 5.25 | 16.67 | 0.00 | 1.79 | 0.00 | 20.83 | 25.00 | 6.67 | 0.00 | 10.00 | 19.44 | ||
recall | 43.09 | 9.09 | 0.00 | 2.27 | 0.00 | 19.32 | 25.00 | 2.67 | 0.00 | 4.55 | 19.44 | ||
Evaluation Results on Dummy EHR dataset | |||||||||||||
tag type | #of tags | scores | Rule | CRF | CRF M | CRF P | CRF M + d | CRF all | LSTM | LSTM M | LSTM P | LSTM M + d | LSTM all |
total | 3017 | F1 | 43.74 | 66.97 | 44.01 | 19.67 | 67.13 | 65.79 | 63.99 | 20.33 | 1.60 | 69.82 | 68.19 |
prec | 42.89 | 66.77 | 67.35 | 56.72 | 67.60 | 68.27 | 68.76 | 26.68 | 2.22 | 72.79 | 80.26 | ||
recall | 44.75 | 67.34 | 33.28 | 12.34 | 66.69 | 63.63 | 60.20 | 17.03 | 1.25 | 67.24 | 60.04 | ||
age | 39 | F1 | 51.13 | 48.46 | 29.35 | 0.00 | 38.87 | 33.82 | 50.00 | 22.38 | 0.00 | 50.00 | 41.67 |
prec | 51.97 | 65.25 | 28.85 | 0.00 | 41.56 | 35.72 | 50.00 | 19.05 | 0.00 | 50.00 | 45.83 | ||
recall | 50.46 | 53.74 | 30.00 | 0.00 | 36.71 | 32.50 | 50.00 | 32.38 | 0.00 | 50.00 | 41.67 | ||
hospital | 170 | F1 | 15.98 | 47.85 | 33.19 | 0.00 | 48.62 | 35.73 | 22.22 | 35.79 | 0.00 | 40.00 | 43.33 |
prec | 10.07 | 53.18 | 38.75 | 0.00 | 44.91 | 35.90 | 28.33 | 34.48 | 0.00 | 37.50 | 45.83 | ||
recall | 39.06 | 43.73 | 29.42 | 0.00 | 53.60 | 37.81 | 29.17 | 37.33 | 0.00 | 43.75 | 41.67 | ||
person | 135 | F1 | 0.00 | 26.96 | 0.00 | 0.00 | 28.36 | 15.48 | 50.00 | 0.00 | 0.00 | 45.83 | 37.50 |
prec | 0.00 | 26.79 | 0.00 | 0.00 | 29.91 | 19.64 | 50.00 | 0.00 | 0.00 | 45.83 | 37.50 | ||
recall | 0.00 | 30.71 | 0.00 | 0.00 | 27.99 | 13.39 | 50.00 | 0.00 | 0.00 | 45.83 | 37.50 | ||
sex | 16 | F1 | 93.75 | 35.92 | 29.17 | 0.00 | 90.08 | 33.93 | 0.00 | 40.00 | 0.00 | 50.00 | 50.00 |
prec | 100.0 | 44.27 | 50.00 | 0.00 | 95.83 | 50.00 | 0.00 | 40.00 | 0.00 | 50.00 | 50.00 | ||
recall | 90.00 | 43.13 | 20.83 | 0.00 | 85.63 | 27.08 | 0.00 | 40.00 | 0.00 | 50.00 | 50.00 | ||
time | 2657 | F1 | 49.48 | 71.28 | 42.14 | 21.20 | 70.60 | 68.33 | 83.93 | 51.97 | 48.89 | 85.70 | 88.20 |
prec | 51.81 | 71.44 | 64.94 | 59.35 | 71.24 | 70.94 | 84.82 | 52.59 | 48.89 | 86.51 | 89.24 | ||
recall | 47.38 | 71.15 | 32.08 | 13.58 | 70.00 | 66.08 | 83.29 | 51.46 | 48.89 | 84.93 | 87.23 | ||