“Help me please … I can’t calm down without laying on the ground and freaking out for a good 20 minutes … Should I get medical help?”
This plea came from a post on the social media site Reddit. The person who posted the question had been having panic attacks for several days after smoking marijuana. Usually, this type of post goes unnoticed by people working in public health. But in a recent experiment, an AI tool was paying attention.
The tool, called Waldo, reviewed more than 430,000 past posts on Reddit forums related to cannabis use. It flagged the post above and over 28,000 others as potentially describing unexpected or harmful side effects. The researchers checked 250 of the posts that Waldo had flagged and verified that 86 percent of them indeed represented problematic experiences with cannabis products, researchers report September 30 in PLOS Digital Health. If this type of scanning became commonplace, the information could help public health workers protect consumers from harmful products.
The beauty of the work, says Richard Lomotey, is that it shows researchers can actually gain information from sources that government agencies, such as the U.S. Centers for Disease Control and Prevention, may not be looking at. The CDC and other agencies take surveys or collect self-reported side effects of illness but do not monitor social media. This is where “people express themselves freely,” says Lomotey, an information technology expert at Penn State.
Many people don’t have access to a doctor or don’t know about the official way to report a bad experience with a product, says John Ayers, a public health researcher at the University of California, San Diego in La Jolla who worked on Waldo. Lots of people share health experiences online. “We need to go where they are,” he says.
Karan Desai, a medical student at the University of Michigan Medical School in Ann Arbor, says the team chose to focus on cannabis products because they are very popular yet largely unregulated. “People in my age demographic, in their 20s, grew up in high school and college with these JUULs, these vapes, these cannabis products,” he says. “I think it’s important for us to know what side effects people are experiencing with using these.”
To prepare Waldo, the team began with a smaller group of 10,000 different Reddit posts about cannabis use. Other researchers had gone through these and identified problematic side effects by hand. Desai and colleagues trained Waldo on a portion of these posts, then tested it on the remaining ones. On this task, the tool outperformed ChatGPT. The general-purpose bot marked 18 times more false positives, indicating posts contained side effects when they didn’t. But it did not outperform the human reviewers.
This all happened before the team’s main experiment, in which Waldo tagged that panic attack post and tens of thousands more.
It remains to be seen whether Waldo would work as well searching for issues related to any kind of drug, vitamin or other product, Lomotey says. AI tools trained on one task may not work as well even on very similar tasks. “We have to be cautious,” he says.
Still, Lomotey imagines a future where tools like Waldo would help keep an eye on social media. This would need to be done carefully, “in an ethical way,” he says. When a person posts about a rare side effect, such tools could flag the issue and pass it on to health officials, with privacy protections in place. He imagines that this could be especially useful in countries that don’t have robust systems in place to monitor and report on drug side effects.
Someday, tools like Waldo might help link people who need help to the public health workers who can provide it. “Even when [side effects] can be rare, when they happen to you, it means all the world,” Ayers says.
Read the full article here
