The Direct Message
Tension: Hospitals are deploying branded AI chatbots positioned as bridges to care, but for the 100 million Americans without a primary care provider, there’s often nothing on the other side of the bridge.
Noise: The industry frames hospital chatbots as safer alternatives to consumer AI, but benchmark accuracy of 95% drops to 33% with real-world patient queries, and no evidence yet shows these tools improve patient outcomes.
Direct Message: A chatbot that mimics the cadence of care without delivering the substance of care doesn’t solve an access crisis. It gives the system permission to stop trying.
Every DMNews article follows The Direct Message methodology.
A freelance graphic designer in Tucson sat on the edge of her bed at 2 a.m. last February, typing her symptoms into a chatbot: chest tightness, tingling left arm, shallow breathing. She hadn’t seen a doctor in three years. Her last employer-sponsored plan ended when the agency she worked for shuttered during a round of client pullbacks. She had no primary care physician, no urgent care she trusted, no one to call at that hour who wouldn’t cost her money she didn’t have. The chatbot told her she was likely experiencing a panic attack. It recommended deep breathing exercises. It suggested she schedule an appointment with a provider. She took two ibuprofen and went back to sleep.
She was fine. This time.
The American healthcare system has a front door problem. Nearly a third of Americans, more than 100 million people, don’t have a primary care provider. The country spends more per capita on healthcare than any peer nation and gets lower life expectancy and more avoidable deaths in return. Into this gap, quietly and with increasing confidence, have stepped the chatbots.
Not the scrappy consumer-facing ones people already use on their phones. The new ones. Branded, hospital-backed, connected to electronic health records. Health systems across the country have begun rolling out their own AI-powered tools, positioning them as the safer, smarter alternative to Googling your symptoms at 2 a.m. As Ars Technica recently reported, Hartford HealthCare has launched PatientGPT, and major health systems are deploying AI-powered chatbots. The pitch is clean: rather than let patients wander into the wilds of unregulated AI, give them a guided experience within the walls of a trusted institution.
The pitch makes sense. The reality is theater.
One in three American adults have already used an AI chatbot for health information, according to a KFF poll. Among those users, 41 percent uploaded personal medical data like test results. And 19 percent said the reason they turned to AI in the first place was because they couldn’t afford care. Another 18 percent said they couldn’t get an appointment or didn’t have a regular provider. Only about a third said the chatbot experience was merely about convenience. The rest were driven by structural failure.
A hardware store manager in rural southwestern Virginia saw his town lose its only hospital four years ago. The nearest clinic is 40 minutes away and books three weeks out. When he developed persistent lower back pain last fall, he asked ChatGPT. It told him he might have a herniated disc, suggested stretches, and recommended he see an orthopedist. Sensible advice. But the nearest orthopedist accepting new patients was in Roanoke, over an hour’s drive. He did the stretches. Six weeks later, the pain was worse. He still hasn’t seen anyone.
People aren’t choosing AI over doctors. They’re choosing AI because doctors aren’t available. This distinction matters enormously, because the hospital systems now deploying branded chatbots are framing the technology as a bridge to care. Healthcare technology executives have emphasized the importance of implementing AI safely within existing health systems that connect to medical records and care teams.
The word “connects” is doing a lot of work in that sentence.
For the chatbot to function as a bridge, there has to be something on the other side. A doctor. An appointment slot. A system that can absorb the patient the AI triages. If none of that exists, the chatbot becomes something else entirely: a very polished dead end that gives the patient the feeling of having been cared for without the substance of actual care. There’s a real risk that the trust people place in chatbots becomes a substitute for the institutional trust that has eroded over decades.
And the theater has a trapdoor. A February study published in Nature Medicine, involving nearly 1,300 participants, found that large language models correctly identified a medical condition about 95 percent of the time when given structured text prompts, the kind of clean scenarios designed to make them look good. But when real patients described their symptoms the way real patients do, in messy, incomplete, anxious language, that accuracy dropped to 33 percent. The models correctly identified appropriate next steps only 43 percent of the time in those real-world interactions, compared with 56 percent when the prompts were structured.
That gap, between the laboratory and the living room, is not a bug to be patched. It is the entire story.
Researchers involved in the study noted that the gap between benchmark scores and real-world performance highlights concerns for AI developers and regulators. Benchmarks are the standardized tests of the AI world. They measure performance under ideal conditions. They don’t capture a 34-year-old freelancer typing with one thumb while trying not to wake her roommate. They don’t capture LLMs struggling with informal patient descriptions of symptoms, such as when someone describes pain in colloquial rather than clinical terms. The 95 percent accuracy that gets cited in press releases and investor decks describes a tool that has never once met a real patient. The 33 percent accuracy describes the tool that actually shows up at 2 a.m.
LLMs are also confidently wrong in ways that human doctors typically aren’t. Research has documented cases of chatbots producing erroneous medical information with complete confidence. The models don’t hesitate. They don’t say “I’m not sure.” They produce language with the cadence and authority of a board-certified physician, even when the content is fiction. For a patient already primed to trust the institution whose name sits above the chatbot’s text box, this confident fabrication can be more dangerous than a blank Google search.
Hartford HealthCare has invested in red-team testing, reportedly reducing high-risk scenario failure rates from 30 percent to 8.5 percent through adversarial testing. That’s a meaningful improvement. But 8.5 percent is not zero. In a system serving hundreds of thousands of patients, an 8.5 percent failure rate on high-risk scenarios means thousands of people receiving dangerous or misleading guidance. And that red-team testing is itself a kind of theater, stress-testing the system against the structured, predictable ways things go wrong while the real failure mode, the one the Nature Medicine study exposed, is the unstructured, unpredictable way real patients actually communicate.
KFF data shows that 58 percent of people who consulted AI about mental health didn’t follow up with a doctor afterward. For physical health concerns, 42 percent didn’t follow up. The chatbot wasn’t a bridge. It was the destination. Clinical reasoning researchers have acknowledged that evidence for chatbots improving patient outcomes remains limited. Three words, “remains limited,” that carry enormous weight when attached to a technology already being deployed at scale.
This pattern of rational but opposite views on AI shows up repeatedly when institutions confront the technology’s limits. Executives see a funnel. Patients see a lifeline. Researchers see a gap. All three are looking at the same tool.
The economics clarify which view prevails. Hospital-branded chatbots aren’t philanthropic projects. They are, as multiple health system executives have acknowledged, patient acquisition tools. A chatbot that triages a patient and then recommends scheduling an appointment at the same health system that built the chatbot is not a neutral advisor. It’s a sales channel with a stethoscope icon. This doesn’t make it evil. It makes it a business, which means it operates under business incentives, which means the question of whether it genuinely improves patient outcomes runs second to the question of whether it drives appointment volume.
There’s a parallel worth noting in how institutions tend to respond to systemic failures. When the post office struggled with aging infrastructure and declining service, the solution often looked like cosmetic fixes layered over structural decay. Institutions under pressure reach for tools that demonstrate activity rather than tools that produce change. A branded chatbot is a visible, marketable, technologically impressive activity. Whether it produces change depends entirely on what happens after the patient closes the chat window.
And the people building these tools, the engineers and clinicians working on red-team testing and safety protocols, aren’t cynics. Many of them are trying to solve a real problem with the tools available to them. The trouble is that the tools available, no matter how sophisticated, address the symptom rather than the disease. Vulnerable populations are particularly exposed, because they are the ones most likely to lack alternatives to the chatbot, the ones for whom 2 a.m. is the only available appointment time, the ones for whom the AI’s confident wrong answer carries the most real-world consequences.
The American healthcare system isn’t broken because people lack information. It’s broken because it was built around a model that assumed most people would have employer-sponsored insurance, a family doctor, and a hospital within driving distance. Tens of millions of Americans no longer fit that model, and no amount of natural language processing changes the underlying math. AI doesn’t create a doctor’s office in rural Virginia. It doesn’t lower emergency room co-pays. It doesn’t add residency slots to a training pipeline that produces too few primary care physicians for the population. What it does is give the system something to point to. Look, we’re innovating. Look, we’re meeting people where they are. Look, we care.
The graphic designer in Tucson is still freelancing. It’s August now. In the six months since her 2 a.m. chest tightness, she’s consulted the chatbot three more times: once for recurring headaches, once for a skin rash that wouldn’t clear, once for the chest tightness again. Each time the chatbot was responsive, empathetic, articulate. Each time it recommended she schedule an appointment with a provider. She still doesn’t have a primary care physician. She looked into marketplace plans during open enrollment but the premiums for a plan with a deductible she could actually meet would have cost more than her car payment. The rash turned out to be contact dermatitis; she figured that out herself, from a Reddit thread, after the chatbot suggested three different possibilities with equal confidence. The headaches come and go. The chest tightness came back last month. She opened the chatbot at 1:47 a.m., read the same recommendation to see a provider, and closed the app. She didn’t even finish the conversation. She already knew what it would say. She already knew she wouldn’t do it. The chatbot had given her the experience of being heard without the reality of being helped, and she had learned, the way patients in a broken system always learn, to stop expecting the difference.
