Artificial intelligence has become one of healthcare’s favorite promises — faster diagnoses, cleaner data, fewer errors, and more efficient systems. But beneath the optimism lies a harder truth: AI doesn’t rise above our biases. It absorbs them. It learns from us. And when our systems are inequitable, our algorithms quietly become inequitable too.
A 2021 commentary in the Canadian Journal of Bioethics examined a case involving a healthcare algorithm called ImpactPro, revealing how AI can unintentionally reproduce racial disparities in care (Sargent, 2021). It’s a striking example of how technology can magnify the very inequities we hope it will solve.
When “Cost” Becomes a Stand‑In for “Care”
ImpactPro was designed to identify patients with complex health needs. But instead of measuring health directly, the system used healthcare spending as a proxy for illness severity. On paper, that seems efficient. In reality, it’s a shortcut built on inequity.
Black patients, because of long‑standing structural barriers, often spend less on healthcare than White patients — not because they are healthier, but because they face more obstacles to accessing care, trusting providers, and receiving equitable treatment (Obermeyer et al., 2019). As Sargent (2021) notes, the algorithm “failed to recommend Black patients to a complex health needs program at the same rate as White patients” (p. 112).
This is a classic example of label bias — when the variable chosen to represent a concept (like “health need”) reflects social inequities rather than clinical reality (Heinrich & Nachum, 2019).
The algorithm wasn’t intentionally discriminatory. It simply learned from the data we gave it.
The Human Layer: Clinicians Aren’t Neutral Either
One of the most revealing findings from the ImpactPro case is that clinicians — the humans meant to correct the algorithm — also showed bias. They were less biased than the original AI, but more biased than the corrected version (Obermeyer et al., 2019).
This matters because it shows that AI bias is not a technical malfunction. It’s a reflection.
Healthcare providers, like all humans, carry implicit biases shaped by training, culture, and experience. Research consistently shows that clinicians hold pro‑White biases and that these biases influence treatment decisions (FitzGerald & Hurst, 2017; Chapman et al., 2013). As Sargent (2021) explains, AI bias “stems from historically biased practices leading to biased datasets, a lack of oversight, as well as bias in practitioners who are overseeing AIs” (p. 112).
AI didn’t invent the problem. It inherited it.
Why Anti‑Bias Training Isn’t Optional Anymore
If AI systems are trained on biased data, and clinicians interpreting AI outputs also carry bias, then the solution cannot be purely technical. It must be human.
Evidence‑based strategies for reducing bias in healthcare include:
- Implicit bias training (Reilly et al., 2013; Gonzalez et al., 2014)
- Education on the history of bias in medicine
- “Individuating” — focusing on the patient as a person, not a stereotype
- “Perspective‑taking” — imagining the patient’s lived experience (Chapman et al., 2013)
- Increasing diversity among healthcare providers, especially Black physicians, who show significantly lower race‑based bias (Chapman et al., 2013)
These practices aren’t new. What’s new is the urgency: AI makes the consequences of bias faster, quieter, and harder to detect.
If clinicians don’t actively counteract bias, AI will amplify it.
AI Ethics Isn’t Just for Engineers — It’s a Clinical Responsibility
One of the most important insights from the ImpactPro case is that frontline healthcare workers are not passive users of AI. They are co‑decision‑makers. They have the authority — and the ethical duty — to question outputs, flag concerns, and advocate for oversight.
This aligns with major AI ethics frameworks like the Montreal Declaration and the EU High‑Level Expert Group on AI, which emphasize equity, transparency, human oversight, and responsibility (Montreal Declaration, 2017; HLEG, 2019).
These principles aren’t abstract. They are clinical obligations.
Healthcare practitioners already have a duty to promote equity in care. AI doesn’t change that duty — it intensifies it.
The Bigger Lesson: AI Won’t Save Us From Ourselves
That 2021 bioethics commentary ends with a line that captures the heart of the issue:
“The biggest potential pitfall with AI is seeing it as a solution to our very human faults, rather than as a tool that reflects what we have done in the past” (Sargent, 2021, p. 115).
AI is not a fix for human bias. It is a magnifier of it.
If we want equitable AI, we must first build equitable healthcare practices. That means confronting the biases that shape our data, our decisions, and our systems. It means investing in trust, representation, and accountability. And it means recognizing that technology cannot be more ethical than the people who design, train, and use it.
AI will not make healthcare fairer on its own. But with intentional, human‑centered oversight, it can become a tool that supports — rather than undermines — equity.