I hurt my left knee over Labor Day weekend, probably doing a lazy breaststroke in the pool. After limping around for a few days, I was able walk normally again. But only for a few minutes, until pain set in.<\/p>\n
Eager for some advice before seeing my primary care doctor, I plugged my symptoms into one of the many <\/span>healthcare chatbots<\/span><\/a> at which <\/span>venture capitalists<\/span><\/a> are throwing large sums of money<\/span>.\u00a0<\/span><\/p>\n The bot and I only chatted for about five minutes, but our conversation felt like a slog. I told the bot my knee made a clicking sound, but there was no way to specify how far I could walk without pain. I didn\u2019t know how to answer the seemingly straightforward question about where I felt pain, and I couldn\u2019t decide whether my knee felt weak or like it might give out.<\/span><\/p>\n Early in the symptom-checking process, I was asked to guess my diagnosis from a peculiar\u00a0list of possibilities: \u201ccellulitis, arthritis, allergic contact dermatitis of the knee, fear of dogs or cancer.\u201d\u00a0<\/strong>After I answered 24 questions, the bot spit out \u201cpatellofemoral pain syndrome” (aka runner\u2019s knee) as a probable diagnosis. It told me to take ibuprofen and <\/span>stretch<\/span><\/a>, and then see a doctor in a few weeks if I still had pain.<\/span><\/p>\n I was going to see my doctor no matter what the bot told me.<\/span> Some experts worry, however, that a vague or inaccurate chatbot diagnosis could endanger a patient\u2019s health or even their life. <\/span>Critics<\/span><\/a> say<\/span><\/a> the algorithm of one of the best-known health chatbots, developed by the British virtual care company Babylon<\/a>,<\/span>\u00a0is dangerously flawed. Dr. David Watkins, a consulting cardiologist with the NHS in London, pointed out that in a test, the bot accurately diagnosed signs of a heart attack in a theoretical male patient but <\/span>misdiagnosed similar symptoms<\/span><\/a> in a theoretical female patient, a pack-a-day smoker in her 60s. The bot said “she” was probably having a panic attack.<\/span><\/p>\n Healthcare bots ask questions to narrow down and rule out the causes of symptoms<\/span>, but they can\u2019t have the kind of organic conversations that often help doctors uncover seemingly unrelated clues about patients\u2019 health, says Dr. Satasuk Joy Bhosai, chief of Digital Health and Strategy at the Duke Clinical Research Institute, part of the Duke University School of Medicine. Bhosai recalls a patient who experienced frequent headaches, one of the most common complaints she hears from patients. The patient noticed his hands were moving slower than usual when he played the piano. It was only when he mentioned this in passing that Bhosai ordered a brain MRI, which revealed a tumor. <\/span>The tumor prompted Bhosai to ask about the patient\u2019s past tobacco use, which eventually led to a lung cancer diagnosis.<\/span><\/p>\n \u201cIf a chatbot asked the patient if he was a smoker, he would\u2019ve said no, and the diagnosis would\u2019ve been missed,\u201d Bhosai says.<\/span><\/p>\n Legal disclaimers for most health chatbots say they shouldn\u2019t be used to diagnose or treat medical conditions. Yet their ability to proffer accurate diagnoses is <\/span>one of their most highly touted <\/span>selling points<\/span><\/a>.<\/span><\/p>\n \u201cI\u2019m not aware of any eHealth chatbot that can be relied upon for diagnosis,\u201d Watkins says. \u201c[And] it\u2019s important to be aware that in the UK, eHealth chatbots are only loosely regulated, with no requirement for validation or regulatory approval. Consequently, there\u2019s a huge variation in accuracy and safety of the chatbots available, ranging from the good to the downright dangerous.\u201d<\/span><\/p>\n Despite concerns that chatbots are too, well, robotic in their interactions with patients, tech giants and startups are <\/span>pursuing contracts<\/span><\/a> to implement them in large health systems and hospitals. These chatbots, also known as \u201cconversational agents” or \u201cpatient-engagement software,\u201d will revolutionize healthcare, proponents say.<\/span><\/p>\n By providing more efficient patient triage and diagnoses, the bots allow healthcare providers to spend more time with the neediest patients. Increased efficiency also helps address doctor burnout and physician shortages by helping patients figure out what they can DIY at home, and by pre-diagnosing patients and directing them to the right specialists. Chatbots might also democratize healthcare by increasing access to nonjudgmental medical advice.\u00a0And unlike humans manning phone lines, chatbots can answer patient questions any time of day.<\/span><\/p>\n Chatbots can remind patients to look out for concerning symptoms, or tell them when it\u2019s time to change their bandages or take medications.<\/span><\/p><\/blockquote>\n \u201cChatbots vary,\u201d says Enid Montague, an expert on human-centered automation in medicine, as well as a professor of computing at DePaul University and an adjunct associate professor at Northwestern University Feinberg School of Medicine.<\/span> \u201cThey can be a simple decision tree or can get more complex by being fed lots of data and using machine learning to make a diagnosis. It could look like a survey or like you\u2019re having a conversation with someone.\u201d<\/span><\/p>\n In the US, chatbots are already interacting with patients. A few hospitals, such as <\/span>Boston Children\u2019s Hospital<\/span><\/a> and the<\/span> Medical College of Wisconsin Health Network, use interactive symptom-checker chatbots. Others, such as <\/span>Northwell Health in New York, <\/span>use chat technology<\/span><\/a> to monitor patients released from the hospital.<\/span><\/p>\n For now, though, most chatbots just schedule appointments or provide information about COVID-19. California healthcare network Sutter Health uses a bot called <\/span>Ada<\/span><\/a> to screen patients for COVID risk factors; the Centers for Disease Control and Prevention has a similar one called <\/span>Clara<\/span><\/a>. In fact, the pandemic has been an opportunity to <\/span>showcase<\/span><\/a> the value<\/span><\/a> of <\/span>chatbots<\/span><\/a>, which have been deployed to field the onslaught of patient COVID questions. <\/span>Developers of the technology<\/span><\/a> hope this will help pave the way for broader adoption of chatbots throughout our healthcare system.<\/span><\/p>\n Experts say the technology is in its infancy and improving rapidly, and will soon become the first stop in patient care. The issue is how to integrate chatbots into care while ensuring they don’t dispense inaccurate or otherwise dangerous information to patients.<\/p>\n Chatbots can be helpful for collecting patient info before a visit and transferring that information to healthcare providers, says Tearsanee Carlisle Davis, a board-certified family nurse practitioner and director of clinical and advanced practice operations at the University of Mississippi Medical Center for Telehealth.<\/p>\n \u201cI think they\u2019re great for pre- and post-surgery care, as the providers and nurses are looking for specific things and the care is pretty standard across the board,\u201d says Davis, who is also a professor at the UMMC School of Nursing. Chatbots can remind patients to look out for concerning symptoms, or tell them when it\u2019s time to change their bandages or take medications.<\/span><\/p>\n A few studies support chatbots\u2019 usefulness in patient after-care. <\/span>A<\/span> 2018 study<\/span><\/a> concluded that text chatbots let nurses make fewer calls to <\/span>chemotherapy patients. A smartphone chatbot also appeared helpful in following up with patients who had surgery for kidney stones, according to a <\/span>2019 study<\/span><\/a>.\u00a0<\/span><\/p>\n A <\/span>study published in July<\/span><\/a> found that people generally liked the COVID chatbots they used. In fact, if they trusted the chatbot, they actually preferred communicating with it over a human, says study author Alan R. Dennis, professor of internet systems at the Indiana University School of Business in Indianapolis.<\/span><\/p>\n \u201cDoctors don\u2019t have time to do two hours of education.”<\/span><\/p><\/blockquote>\n Weill Cornell Medicine, in New York, uses its chatbot Hyro as an administrative aid and symptom-checker. Hyro can assist patients with simple tasks, such as finding a cardiologist in the health system who speaks Spanish, says Hyro content marketing specialist Ziv Gidron.\u00a0<\/span><\/p>\n Chatbots can also help doctors spend less time educating patients. Researchers at the Boston University Medical Center created <\/span>Gabby<\/span><\/a>, a video \u201chost\u201d who walks patients through <\/span>important preconception care<\/span><\/a>. Gabby\u2019s purpose is to improve the health of Black mothers and their babies, who experience <\/span>disproportionately poor health outcomes<\/span><\/a> compared to white women.<\/span><\/p>\n \u201cDoctors don\u2019t have time to do two hours of education,\u201d says Gabby designer Dr. Brian Jack, d<\/span>irector of the Center for Health System Design and Implementation at Boston University\u2019s <\/span>Institute for Health System Innovation and Policy and a professor at the BU Scho<\/span>ol of Medicine. Gabby is more engaging than informational pamphlets, which, Jack says, patients typically don\u2019t read.<\/span><\/p>\n Chatbots might also play a role in mental healthcare. For example, University of Illinois Chicago researchers<\/span> are studying<\/span><\/a> whether Lumen, a voice-activated AI program, can help patients manage moderate depression or anxiety. And a<\/span> 2017 study<\/span><\/a> suggests that<\/span> Woebot<\/span><\/a>, a free cognitive behavioral therapy app, helped relieve symptoms of depression in subjects, who were mostly young white women.\u00a0 <\/span><\/p>\n Researchers at the University of Pennsylvania acknowledged Woebot\u2019s success in a <\/span>recent opinion<\/span><\/a> published in\u00a0<\/span>JAMA,<\/span><\/i> but study authors also warn that conversational agents are \u201cnot yet mature enough to reliably respond to patient statements,\u201d even when those statements signal self-harm by mentioning suicidal thoughts. Study authors question whether it\u2019s appropriate for chatbots to offer diagnoses and treatment options when most companies\u2019 research, if they\u2019ve done any, is based on data, not real patients.<\/span><\/p>\n Other researchers have raised <\/span>similar concerns<\/span><\/a>. In a 2019 <\/span>BMC Medicine <\/span><\/i>paper, study authors said it’s critical that \u201cthe minimal requirements used to make a digital health technology available to the public are not mistaken for a product that has passed rigorous testing or demonstrated real world therapeutic value.\u201d<\/span><\/p>\n Because tech companies\u2019 algorithms are considered proprietary, study authors noted, they aren\u2019t required to reveal how they work or whether they\u2019re trained using representative data (rather than actual patients).<\/span><\/p>\n A \u201cmove fast and break things\u201d approach can be particularly risky in healthcare.<\/span><\/p><\/blockquote>\n These are common issues in tech, where innovation is often prized over caution. But a \u201cmove fast and break things\u201d approach can be particularly risky in healthcare. Because <\/span>the Food and Drug Administration does not consider<\/span><\/a> health chatbots \u201cmedical devices,\u201d they don’t require FDA approval.<\/span><\/p>\n The FDA told Dr. Vipindas Chengat, CEO of patient-engagement software company MayaMD AI<\/a>, the company’s chatbot would only be considered a medical device if it gave ‘specific, granular recommendations, such as decreasing medical dosage.” That saves companies from conducting extensive clinical research.<\/p>\n Even if companies are willing to do <\/span>their due diligence, no guidelines have been established for <\/span>how<\/span><\/i> testing should be done.\u00a0<\/span><\/p>\n \u201cIt\u2019s challenging for startups to develop the evidence piece because they\u2019re under pressure to scale quickly,\u201d Bhosai says. \u201cStudies can be off their radar. And sometimes they don’t want results and are not interested in having an academic center look at data they already have.\u201d<\/span><\/p>\n Bhosai provides clinical guidance for companies who want to responsibly test their products and evaluate feedback. Hospitals and healthcare providers generally want to see data supporting efficacy before they adopt e-health tools, she notes.<\/span><\/p>\n Another issue as healthcare chatbots become more ubiquitous, Bhosai says, is that software developers aren\u2019t required to make programs inclusive.<\/span><\/p>\n \u201cThere has to be some work toward managing the digital divide in health disparities,\u201d she says. \u201cI worry about the patient who can’t read or who can\u2019t speak English, or older folks. Even if they use the internet, they might not be able to see well and read questions online. People of different [backgrounds] also might communicate differently; the way questions are asked matters.\u201d<\/span><\/p>\n Systems are only as valid as the data that goes into them, Montague adds. “B<\/span>iases can be built into the design, such as the assumption that everyone lives in a wealthy [neighborhood] and has a life expectancy of 90 years old,” she says. “We need good data that represents everyone and interdisciplinary teams of people well-versed in health and racial disparities actually happening in communities. We\u2019re not really at that stage.\u201d<\/span><\/p>\n Researchers and doctors are calling for greater transparency and responsibility from health chatbot makers before they’re unleashed on patients. Two new papers, published in <\/span>The Lancet<\/span><\/i><\/a> and in <\/span>Bio Integration<\/span><\/i><\/a>, <\/span>discuss how to integrate\u00a0AI into medical practice responsibly and ethically. In addition, the authors of the <\/span>BMC Medicine<\/i><\/a> paper mentioned above developed a Digital Health Decision-Making Framework and Checklist<\/a> that covers patient privacy issues, responsible data management, ethical risk-benefit analyses and considerations to make tech accessible to diverse populations. The checklist was originally intended for researchers, but study authors say it would help developers and clinicians as well.<\/p>\nThe “first stop in patient care”<\/h3>\n
How bots can improve care<\/h3>\n
Bot caveats<\/h3>\n