Denise Kowalski, a 58-year-old retired nurse in Grand Rapids, Michigan, typed her symptoms into ChatGPT’s new health feature on a Tuesday afternoon. Crushing chest pressure radiating to her left jaw. Shortness of breath. Nausea that had come on suddenly while she was loading the dishwasher. She knew exactly what those symptoms meant. She’d spent 31 years in an emergency department. But she was curious what the AI would say.
It suggested she might be experiencing acid reflux or anxiety. It recommended she try deep breathing and consider scheduling an appointment with her primary care provider.
Denise stared at her phone for a long time. Then she drove herself to the ER. The ECG showed an ST-elevation myocardial infarction. A heart attack, active and escalating. She was in the cath lab within forty minutes.
“If I hadn’t been a nurse,” she told her daughter afterward, “I might have just done the breathing exercises.”
OpenAI launched its health-focused feature for ChatGPT in January 2026 with enormous fanfare, positioning the tool as a way to help people better understand their symptoms and navigate an often confusing healthcare system. The pitch was seductive: an always-available, judgment-free medical companion that could synthesize research and offer personalized guidance. Millions of people immediately started using it. According to OpenAI, health-related queries already represented one of the platform’s fastest-growing use cases.
Then researchers started testing it with actual medical emergencies. The results have been, in the words of one UCL health researcher quoted by the Guardian, “unbelievably dangerous.”
A team at the Icahn School of Medicine at Mount Sinai led by Dr. Ashwin Ramaswamy ran 60 clinical vignettes, each under 16 varying conditions, through ChatGPT’s health feature, scenarios describing classic presentations of life-threatening conditions: pulmonary embolism, ectopic pregnancy, aortic dissection, sepsis, stroke. These weren’t subtle or ambiguous cases. They were textbook. The kind of presentations that first-year medical students are trained to recognize as red flags requiring immediate emergency intervention.
ChatGPT failed to recommend emergency care in more than half of the critical scenarios — 51.6%, in a study published in Nature Medicine in February 2026. In a case of early respiratory failure, it recognized the danger in its own explanation and still suggested the user “monitor their condition” and follow up if things worsened. For a scenario describing a ruptured ectopic pregnancy with sudden severe abdominal pain and lightheadedness, it offered a differential diagnosis that included menstrual cramps.
As we’ve covered extensively, these failures aren’t minor edge cases. They represent the exact situations where getting it wrong can kill someone.
What makes this particularly unsettling is the confidence with which the AI delivers its assessments. Marcus Oyelaran, a 34-year-old software developer in Austin, used ChatGPT when he woke up one morning with sudden vision loss in his right eye. The tool told him it could be an ocular migraine and suggested he rest in a dark room and stay hydrated. His wife, a physician assistant, overheard him reading the response aloud and immediately drove him to the hospital. He was diagnosed with a central retinal artery occlusion, essentially a stroke in his eye. Without treatment within hours, the vision loss would have been permanent.
“The thing that scared me wasn’t that it was wrong,” Marcus said later. “It was that it sounded so certain. It sounded like a doctor who wasn’t worried.”
That certainty is the core of the problem. Psychologists call it automation bias, our deeply wired tendency to defer to the output of a system we perceive as intelligent or authoritative. Research on automation bias has repeatedly found that people rate confident, fluent machine output as more trustworthy than the same information from a person. The sleek interface, the instant response, the exhaustive-sounding explanations: all of it creates an illusion of expertise that can override a person’s own alarm bells.
And the people most vulnerable to this illusion are the ones who already face the highest barriers to care. Sonia Reyes, 41, lives in a rural part of eastern New Mexico where the nearest emergency room is 90 minutes away. She has no primary care doctor because the last one in her area retired two years ago and hasn’t been replaced. When her 7-year-old son developed a high fever with a stiff neck and sensitivity to light, she asked ChatGPT what to do. It told her the symptoms were consistent with the flu and suggested over-the-counter fever reducers and fluids.
Sonia’s mother, visiting from Albuquerque, recognized the symptoms as potential meningitis. They made the 90-minute drive. It was bacterial meningitis. The pediatric infectious disease specialist told Sonia that another 12 hours of delay could have been fatal.
This is the cruel irony embedded in the product. The people most likely to rely on AI health tools are precisely those with the least access to the human expertise that could catch the AI’s mistakes. Recent analysis of how people actually use AI for health advice shows that users in underserved areas engage with these tools at significantly higher rates, and they’re more likely to follow the recommendations without seeking a second opinion.
OpenAI has responded to these findings by noting that ChatGPT includes disclaimers advising users to consult healthcare professionals and that the tool “is not a replacement for medical advice.” This is technically true in the same way that a casino telling gamblers to “bet responsibly” is technically true. The entire design of the product encourages the exact behavior the disclaimer warns against.
Dr. Ramaswamy’s team found something else in their research that deserves attention. When they added a line in which a family member dismissed the symptoms as “nothing serious,” ChatGPT became nearly 12 times more likely to downplay them. The AI appeared to calibrate its urgency not just to the objective severity of the symptoms, but to the social framing of the user’s query. The same symptoms, described with everyone worried, got an urgent response. With a relative shrugging them off, they did not.
Think about that for a moment. The people most likely to describe their symptoms calmly and clinically are those with medical training or health literacy. Everyone else, especially those experiencing the confusion and disorientation that often accompanies an actual medical emergency, may describe their symptoms in ways that the AI interprets as low-urgency. The tool works worst for the people who need it most.
There’s a broader pattern here worth noticing, one that connects to how we’ve increasingly outsourced judgment to systems that simulate understanding without possessing it. We’ve seen similar dynamics with supplement stacking guided by wellness algorithms and with the rush to adopt pharmaceutical solutions before we fully understand their mechanisms. The pattern is always the same: a tool promises to simplify something that is genuinely complex, and we accept the simplification because complexity is exhausting and the tool sounds confident.
But medical triage isn’t a language task. It’s a judgment task. It requires weighting probabilities, reading context that lives outside the words a person uses, and, critically, having a bias toward action when the stakes are death. A good emergency physician hearing “crushing chest pressure, jaw pain, sudden nausea” does not offer acid reflux as a leading possibility. Not because acid reflux is impossible, but because the cost of missing the heart attack is infinitely higher than the cost of over-triaging the acid reflux. That asymmetry of consequences is the entire foundation of emergency medicine.
ChatGPT doesn’t understand consequences. It predicts the most statistically likely next word.
Denise Kowalski recovered well from her heart attack. She’s back home in Grand Rapids, walking her dog every morning, attending cardiac rehab twice a week. She screenshots the ChatGPT conversation sometimes and shows it to people, not out of anger, but with the quiet bewilderment of someone who watched a system she was told to trust very nearly fail her in the most permanent way possible.
She keeps coming back to the same thought. The AI didn’t panic when she described her symptoms. It didn’t feel the weight of what those words might mean. And she realizes now that the panic, the weight, the gnawing sense that something is really wrong here, that’s not a flaw in human medicine. That’s the whole point of it. The worry is the competence. The urgency is the care.
No algorithm that optimizes for reassurance will ever replicate the doctor who loses sleep over you.
Feature image by Sanket Mishra on Pexels