Automated SNOMED CT concept and attribute relationship detection through a web-based implementation of cTAKES

Table 7 Evaluation outcomes of the different implementations and datasets

				Named-entity recognition					Attribute relation detection
Implementation	Set	Approach	Outcome	Lung cancer^a		Non-small cell lung cancer^b		Recurrence^c	Recurrent lung cancer^d
Implementation	Set	Approach	Outcome	Present	Absent	Present	Absent	Present	Present	Absent
1	Development set (n = 30)	Relaxed	Detected	15	1	11	1	0
cTAKES with UMLS 2011		Relaxed	Not detected	12	2	6	12	15
		Strict	Detected	15	1	11	1
		Strict	Not detected	8	6	6	12
	Test set (n = 68)	Relaxed	Detected	35	0	29	1	0
		Relaxed	Not detected	16	17	7	31	30
		Strict	Detected	33	2	29	1
		Strict	Not detected	7	26	2	36
2	Development set (n = 30)	Relaxed	Detected	10	0	5	0	0
cTAKES with UMLS 2016		Relaxed	Not detected	17	3	12	13	15
		Strict	Detected	10	0	5	0
		Strict	Not detected	13	7	12	13
	Test set (n = 68)	Relaxed	Detected	29	0	18	1	0
		Relaxed	Not detected	22	17	18	31	30
		Strict	Detected	29	0	18	1
		Strict	Not detected	11	28	13	36
3	Development set (n = 30)	Relaxed	Detected	23	0	13	0	15
cTAKES with UMLS 2016 and custom dictionary		Relaxed	Not detected	4	3	4	13	0
		Strict	Detected	23	0	13	0
		Strict	Not detected	0	7	4	13
	Test set (n = 68)	Relaxed	Detected	36	0	22	1	30
		Relaxed	Not detected	15	17	14	31	0
		Strict	Detected	36	0	22	1
		Strict	Not detected	4	28	9	36
4	Development set (n = 30)	Relaxed	Detected	23	0	17	0	15	6	0
DIRECT cTAKES with UMLS 2016, custom dictionary, and post-processing		Relaxed	Not detected	4	3	0	13	0	0	10
		Strict	Detected	23	0	17	0
		Strict	Not detected	0	7	0	13
	Test set (n = 68)	Relaxed	Detected	36	0	28	1	29	9	0
		Relaxed	Not detected	15	17	8	31	1	3	18
		Strict	Detected	36	0	28	1
		Strict	Not detected	4	28	3	36

^aSNOMED CT concept 93880001 | Primary malignant neoplasm of lung (disorder)
^bSNOMED CT concept 254637007 | Non-small cell lung cancer (disorder)
^cSNOMED CT concept 255227004 | Recurrent (qualifier value)
^dRelationship between three SNOMED CT concepts: 93880001 | Primary malignant neoplasm of lung (disorder) |: 263502005 | Clinical course (attribute) | = 255227004 | Recurrent (qualifier value)

ISSN: 2041-1480