Text Mining for Diverse Review Topics: A Prospective Study Comparing Search Strategies Developed with and without Text Mining Tools

Objectives: To compare the costs and benefits of searches with and without text-mining tools (TMT):
1) Do TMT decrease the time spent developing strategies?
2) Do TMT identify groups of records that can be safely excluded?
3) Do TMT improve search performance (specificity, sensitivity)?
4) How does the performance of TMT for developing strategies compare for simple or complex review topics?

Methods: In this prospective study, we recruited nine systematic review projects, classifying their topics as simple or complex. Each project's information specialist used conventional methods to create a MEDLINE search strategy and another paired information specialist independently created a MEDLINE search strategy using text-mining tools. Text-mining searches were created using freely-available TMT to ensure replication of our methods and relevancy of our findings to all review producers. We collected search results for both MEDLINE strategies, coded and removed duplicates, and sent the citations to the review team for screening. We used the final list of included studies to calculate the sensitivity, specificity, precision, and Number Needed to Read (NNR) for both MEDLINE strategies. We tracked time spent by information specialists to conduct each task in their search development process. Simple and complex topics were analyzed separately to allow comparison.

Results: UP searches were more sensitive (92% (95% CI, 85%-99%) than TMT searches (84.9% (95% CI, 74.4%-95.4%). The mean number-needed-to-read was 83 (SD 34) for UP and 90 (SD 68) for TMT. Model search strategy development took UP librarians 12 hours (SD 8) and TMT librarians 5 hours (SD 2). TMTs did not improve identification of excludable records.

Conclusions: Across all reviews and for simple SR topics, TMT searches were generally less sensitive and reduced time spent in strategy development than UP searches. For complex SR topics, TMT searches were also less sensitive, but did identify includable studies not found by the UP search.