Skip to main content

Table 3 Precision, recall and calculated F-scores from the evaluation outcomes

From: Automated SNOMED CT concept and attribute relationship detection through a web-based implementation of cTAKES

  Development set (n = 30) Test set (n = 68)
Implementation Algorithm Concept Approach Precision Recall F-score Precision Recall F-score
UMLS2011 Named-entity recognition Lung cancera Implied 0.938 0.556 0.698 1.000 0.686 0.814
cTAKES with UMLS 2011 Strict 0.938 0.652 0.769 0.943 0.825 0.880
Non-small cell lung cancerb Implied 0.917 0.647 0.759 0.967 0.806 0.879
Strict 0.917 0.647 0.759 0.967 0.935 0.951
Recurrencec   0.000 0.000 0.000 0.000 0.000 0.000
UMLS2016 Named-entity recognition Lung cancera Implied 1.000 0.370 0.541 1.000 0.569 0.725
cTAKES with UMLS 2016 Strict 1.000 0.435 0.606 1.000 0.725 0.841
Non-small cell lung cancerb Implied 1.000 0.294 0.455 0.947 0.500 0.655
Strict 1.000 0.294 0.455 0.947 0.581 0.720
Recurrencec   0.000 0.000 0.000 0.000 0.000 0.000
UMLS2016 + Dict. Named-entity recognition Lung cancera Implied 1.000 0.852 0.920 1.000 0.706 0.828
cTAKES with UMLS 2016 and custom dictionary Strict 1.000 1.000 1.000 1.000 0.900 0.947
Non-small cell lung cancerb Implied 1.000 0.765 0.867 0.957 0.611 0.746
Strict 1.000 0.765 0.867 0.957 0.710 0.815
Recurrencec   1.000 1.000 1.000 0.882 1.000 0.938
DIRECT Named-entity recognition Lung cancera Implied 1.000 0.852 0.920 1.000 0.706 0.828
cTAKES with UMLS 2016, custom dictionary, and additional processing Strict 1.000 1.000 1.000 1.000 0.900 0.947
Non-small cell lung cancerb Implied 1.000 1.000 1.000 0.966 0.778 0.862
Strict 1.000 1.000 1.000 0.966 0.903 0.933
Recurrencec   1.000 1.000 1.000 0.879 0.967 0.921
Attribute relationship detection Recurrent lung cancerd   1.000 1.000 1.000 1.000 0.750 0.857
  1. aSNOMED CT concept 93880001 | Primary malignant neoplasm of lung (disorder)
  2. bSNOMED CT concept 254637007 | Non-small cell lung cancer (disorder)
  3. cSNOMED CT concept 255227004 | Recurrent (qualifier value)
  4. dRelationship between three SNOMED CT concepts: 93880001 | Primary malignant neoplasm of lung (disorder) |: 263502005 | Clinical course (attribute) | = 255227004 | Recurrent (qualifier value)