Online symptom-checkers are often wrong

Online symptom checkers often misdiagnose patients’ problems, often encouraging people to seek care for minor issues that don’t need immediate attention and other times incorrectly telling people with true emergencies that treatment can wait, a U.K. study suggests.

Researchers tested 23 online and mobile apps used by millions of people who are trying to find out if their symptoms are serious and what might make them feel better. The apps were imperfect at best, offering the correct diagnosis on the first try only about a third of the time.

For triage - assessing the urgency of the problem - the apps were too cautious in situations requiring only self-care: only 33 percent of the time, on average, were patients appropriately advised not to go to the doctor.

At the other extreme, symptom checkers typically missed the severity of the situation in one of every five cases requiring emergency treatment.

Overall, the computer programs offered accurate triage advice for 57 percent of the standardized scenarios that were used in the researchers’ tests.

“The risk is that people will be told to get care when they didn’t need it and bear the costs and inconvenience, or they will be told not to seek care when they have a life-threatening problem,” senior author Dr. Ateev Mehrotra, a health policy researcher at Harvard Medical School in Boston, said by email.

Because patients may not get much useful information from a long list of possible diagnoses, the researchers rated the symptom-checkers based on whether the programs spit out the right answer first, or somewhere lower down on a list of up to 20 possible alternative diagnoses.

The software listed the right diagnosis first in 24 percent of emergencies on average, and for 40 percent of non-urgent cases. Accuracy was better for common than for rare diagnoses.

The app that did best at giving the correct diagnosis on the first try was DocResponse.com, at 50 percent.

Accuracy was better for the systems that used Schmitt or Thompson nurse triage protocols, standard diagnostic tools used by clinicians to provide advice over the phone. With these nurse triage protocols, 72 percent of patients got the appropriate guidance on the urgency of their situation, compared with 55 percent without the protocols.

One limitation of the study is that it used specific clinical language to describe medical conditions in the test vignettes, which may not provide an accurate reflection of how the symptom checkers would perform for patients using nonclinical terms to describe their conditions, the authors acknowledge in the British Medical Journal.

More research is needed to help build more accurate, safer and more effective tools, and symptom checkers are unlikely to completely replace the role of humans in diagnosis, particularly for patients without Internet access, said Jeremy Wyatt, an e-health researcher at the Leeds Institute of Health Sciences in the U.K., in an editorial.

In some situations, symptom checkers are never a good idea, Wyatt said by email. Patients who think they have something seriously wrong – such as a very high fever, inability to stand, rapid breathing, severe pain or bleeding – shouldn’t ask a computer whether they need immediate help.

“Don’t waste your time surfing, call 911,” Wyatt said.

If it’s less urgent, patients may be able to spare the time to use these online and mobile tools, Wyatt said. Patients should look for reliable symptom checkers such as DocResponse.com or FamilyDoctor.org, or one that uses the Schmitt or Thompson nurse triage protocols, he added.

Non-urgent patients “can probably afford to spend a few minutes checking symptoms online,” Wyatt said. “However, avoid just putting your symptoms into Google; use a reliable online symptom checking tool.”