Table 3 Results of the recommendation for a random draw, for the two baselines and for RecSOI according to the heuristics explained in the Evaluation heuristics section and the performance metrics MAP\(_\exists\)@5, MAP\(_\exists\)@10 and MAP\(_\exists\)@20
From: RecSOI: recommending research directions using statements of ignorance
Metric | Method | First-author heuristic | Co-authors heuristic | Concepts heuristic |
|---|---|---|---|---|
MAP\(_\exists\)@5 | Random | 0% \(\pm 0.4\%\) | 0% \(\pm 3.1\%\) | 0% \(\pm 0.4\%\) |
sentence-BERT | 31.2% \(\pm 4\%\) | 13.6% \(\pm 8.8\%\) | 65.3% \(\pm 4.2\%\) | |
TF-IDF+LR | 23% \(\pm 3.7\%\) | 6.8% \(\pm 6.8\%\) | 43.3% \(\pm 4.3\%\) | |
RecSOI | 34.6% \(\pm 4.2\%\) | 10.2% \(\pm 7.9\%\) | 75.6% \(\pm 3.8\%\) | |
MAP\(_\exists\)@10 | Random | 0% \(\pm 0.4\%\) | 0% \(\pm 3.1\%\) | 15.9% \(\pm 3.2\%\) |
sentence-BERT | 37.8% \(\pm 4.2\%\) | 15.3% \(\pm 9.2\%\) | 74.4% \(\pm 3.8\%\) | |
TF-IDF+LR | 28.2% \(\pm 3.9\%\) | 8.5% \(\pm 7.4\%\) | 58.1% \(\pm 4.3\%\) | |
RecSOI | 42% \(\pm 4.3\%\) | 18.6% \(\pm 9.8\%\) | 84.3% \(\pm 3.2\%\) | |
MAP\(_\exists\)@20 | Random | 0% \(\pm 0.4\%\) | 0% \(\pm 3.1\%\) | 46% \(\pm 4.4\%\) |
sentence-BERT | 45.6% \(\pm 4.4\%\) | 23.7% \(\pm 10.6\%\) | 82.3% \(\pm 3.4\%\) | |
TF-IDF+LR | 38.2% \(\pm 4.2\%\) | 11.9% \(\pm 8.3\%\) | 74.6% \(\pm 3.8\%\) | |
RecSOI | 46.8% \(\pm 4.4\%\) | 23.7% \(\pm 10.6\%\) | 90.9% \(\pm 2.5\%\) |