Skip to main content

Table 6 Evaluation results for each tag and in total, for different methods (rule, CRF, LSTM) and different evaluation datasets (MedNLP, dummy EHR, and pathology reports). M, d, and P respectively denote training data of MedNLP, dummy EHR, and Pathology reports; M + d denotes that training data consist of MedNLP+dummy EHR, all stands for all of these three datasets; other machine learning methods use the target evaluation dataset as its training data. In each cell, F1-score, precision, and recall are shown (in values multiplied by 100). The best scores for each tag type for each evaluation metric are presented in bold typeface. All evaluations were done by four-fold cross validations

From: De-identifying free text of Japanese electronic health records

Evaluation Results on MedNLP dataset
tag type #of tags scores Rule CRF CRF
d
CRF
P
CRF
M + d
CRF
all
LSTM LSTM
d
LSTM
P
LSTM
M + d
LSTM
all
total 490 F1 84.23 82.62 43.85 0.71 26.40 67.34 83.07 41.26 0.43 67.35 57.03
prec 78.90 85.63 46.20 2.50 21.51 66.54 81.33 41.07 0.48 66.98 57.94
recall 90.42 79.95 42.33 0.41 59.76 68.38 86.12 41.57 0.38 68.17 56.34
age 56 F1 93.43 71.12 30.00 0.00 32.55 53.04 95.83 71.11 0.00 84.72 87.50
prec 96.00 78.24 37.50 0.00 26.93 56.85 95.83 71.11 0.00 84.72 87.50
recall 91.16 65.47 28.13 0.00 46.05 50.00 95.83 71.11 0.00 84.72 87.50
hospital 75 F1 84.73 87.09 43.25 0.00 26.02 70.04 66.67 13.33 13.89 66.67 41.67
prec 80.75 93.52 66.67 0.00 20.55 91.67 75.00 11.11 10.67 70.83 45.83
recall 89.90 81.71 27.50 0.00 53.06 60.42 62.50 16.67 20.00 63.89 38.89
person 0   N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A
sex 4 F1 50.00 16.67 16.67 0.00 14.65 25.00 0.00 20.00 0.00 25.00 25.00
prec 50.00 25.00 12.50 0.00 8.68 25.00 0.00 20.00 0.00 25.00 25.00
recall 50.00 12.50 25.00 0.00 50.00 25.00 0.00 20.00 0.00 25.00 25.00
time 355 F1 50.00 16.67 47.43 0.98 14.65 70.57 96.14 67.22 42.98 89.78 82.67
prec 50.00 25.00 45.16 2.50 8.68 65.46 95.00 66.26 39.46 88.68 81.53
recall 50.00 12.50 50.19 0.61 50.00 76.50 97.41 68.30 47.94 91.00 82.67
Evaluation Results on Pathology Report dataset
tag type #of tags scores Rule CRF CRF
M
CRF
d
CRF
M + d
CRF
all
LSTM LSTM
M
LSTM
d
LSTM
M + d
LSTM
all
all 71 F1 13.97 74.26 0.00 0.62 1.45 57.63 81.67 0.00 0.00 1.45 81.25
prec 8.65 86.72 0.00 1.47 10.00 64.98 86.88 0.00 0.00 10.00 82.48
recall 43.33 65.16 0.00 0.39 0.78 54.06 78.84 0.00 0.00 0.78 80.15
age 0   N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A
hospital 31 F1 31.19 0.00 0.00 0.00 0.00 0.00 25.00 0.00 13.33 0.00 58.33
prec 26.47 0.00 0.00 0.00 0.00 0.00 25.00 0.00 13.33 0.00 58.33
recall 41.28 0.00 0.00 0.00 0.000 0.00 25.00 0.00 13.33 0.00 58.33
person 224 F1 0.00 91.08 0.00 0.00 6.25 71.31 95.19 0.00 0.00 0.00 95.83
prec 0.00 95.83 0.00 0.00 10.00 74.79 95.19 0.00 0.00 0.00 95.83
recall 0.00 87.21 0.00 0.00 4.55 69.63 95.19 0.00 0.00 0.00 95.83
sex 0   N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A
time 40 F1 9.25 10.57 0.00 2.00 0.00 18.82 25.00 3.81 0.00 6.25 19.44
prec 5.25 16.67 0.00 1.79 0.00 20.83 25.00 6.67 0.00 10.00 19.44
recall 43.09 9.09 0.00 2.27 0.00 19.32 25.00 2.67 0.00 4.55 19.44
Evaluation Results on Dummy EHR dataset
tag type #of tags scores Rule CRF CRF
M
CRF
P
CRF
M + d
CRF
all
LSTM LSTM
M
LSTM
P
LSTM
M + d
LSTM
all
total 3017 F1 43.74 66.97 44.01 19.67 67.13 65.79 63.99 20.33 1.60 69.82 68.19
prec 42.89 66.77 67.35 56.72 67.60 68.27 68.76 26.68 2.22 72.79 80.26
recall 44.75 67.34 33.28 12.34 66.69 63.63 60.20 17.03 1.25 67.24 60.04
age 39 F1 51.13 48.46 29.35 0.00 38.87 33.82 50.00 22.38 0.00 50.00 41.67
prec 51.97 65.25 28.85 0.00 41.56 35.72 50.00 19.05 0.00 50.00 45.83
recall 50.46 53.74 30.00 0.00 36.71 32.50 50.00 32.38 0.00 50.00 41.67
hospital 170 F1 15.98 47.85 33.19 0.00 48.62 35.73 22.22 35.79 0.00 40.00 43.33
prec 10.07 53.18 38.75 0.00 44.91 35.90 28.33 34.48 0.00 37.50 45.83
recall 39.06 43.73 29.42 0.00 53.60 37.81 29.17 37.33 0.00 43.75 41.67
person 135 F1 0.00 26.96 0.00 0.00 28.36 15.48 50.00 0.00 0.00 45.83 37.50
prec 0.00 26.79 0.00 0.00 29.91 19.64 50.00 0.00 0.00 45.83 37.50
recall 0.00 30.71 0.00 0.00 27.99 13.39 50.00 0.00 0.00 45.83 37.50
sex 16 F1 93.75 35.92 29.17 0.00 90.08 33.93 0.00 40.00 0.00 50.00 50.00
prec 100.0 44.27 50.00 0.00 95.83 50.00 0.00 40.00 0.00 50.00 50.00
recall 90.00 43.13 20.83 0.00 85.63 27.08 0.00 40.00 0.00 50.00 50.00
time 2657 F1 49.48 71.28 42.14 21.20 70.60 68.33 83.93 51.97 48.89 85.70 88.20
prec 51.81 71.44 64.94 59.35 71.24 70.94 84.82 52.59 48.89 86.51 89.24
recall 47.38 71.15 32.08 13.58 70.00 66.08 83.29 51.46 48.89 84.93 87.23