Bias, safety, and accountability gaps persist in deployed healthcare AI systems

The rapid rollout of artificial intelligence (AI) in healthcare is exposing weaknesses in how digital tools are governed once they reach the bedside. While AI systems are increasingly used in diagnostics, decision support, and hospital operations, new research suggests that many continue to pose unmanaged risks to patient safety and equity. According to a study, these risks persist not because ethics are absent, but because governance mechanisms are poorly implemented in practice.

Published in Sci, the study Governing Healthcare AI in the Real World: How Fairness, Transparency, and Human Oversight Can Coexist reviews international evidence showing that bias, opacity, and accountability gaps often emerge after deployment, when oversight is weakest.

Bias and transparency risks persist after deployment

Bias in healthcare AI is not a static problem that can be solved at the design stage. Instead, bias evolves over time as systems encounter new patient populations, shifting clinical practices, and changing data inputs. Models trained on historical datasets may initially perform well, but their accuracy and fairness can degrade after deployment, particularly for underrepresented or vulnerable groups. The authors emphasize that this dynamic nature of bias is often overlooked in governance frameworks that focus narrowly on pre-deployment validation.

In real-world healthcare settings, patient demographics, disease prevalence, and care pathways are rarely stable. When AI systems are deployed without continuous monitoring, performance disparities can emerge unnoticed. The study highlights that many healthcare institutions lack the infrastructure, expertise, or contractual authority to audit AI systems after purchase. This creates a governance blind spot where tools continue to influence clinical decisions even as their reliability changes.

Transparency presents a parallel challenge. While explainability has become a central concept in healthcare AI ethics, the authors find that explainability tools are frequently misaligned with actual clinical and regulatory needs. Explanations generated by AI systems may satisfy technical benchmarks but fail to provide meaningful insight for clinicians who must justify decisions to patients or regulators. At the same time, transparency mechanisms designed for developers often do not translate into accountability mechanisms for healthcare institutions.

The study argues that transparency must be differentiated by audience and purpose. Clinicians require explanations that support clinical reasoning and shared decision-making. Patients need understandable information about how AI influences their care. Regulators and auditors require traceability and documentation that demonstrate compliance and risk management. Treating transparency as a single, uniform requirement, the authors conclude, leads to systems that appear explainable in theory but opaque in practice.

Safety, privacy, and the accountability gap

Safety has been identified as a persistent and under-addressed risk in healthcare AI governance. Safety failures, the authors note, often arise not from catastrophic system errors but from gradual mismatches between AI outputs and clinical workflows. Changes in staffing, protocols, or patient populations can alter how AI recommendations are interpreted and acted upon. Without clear processes to reassess and recalibrate systems, these mismatches can accumulate into clinically significant harm.

The study points out that responsibility for AI safety is frequently unclear. Developers may argue that systems perform as designed, while healthcare providers assume that regulatory approval guarantees ongoing safety. This diffusion of responsibility creates what the authors describe as an accountability gap. When adverse outcomes occur, it is often difficult to determine who has the authority and obligation to intervene, suspend, or withdraw an AI system from use.

Privacy risks further complicate this landscape. The increasing use of adaptive and generative models raises concerns about data leakage, memorization, and unintended secondary use of sensitive health information. Traditional anonymization techniques, the authors argue, are no longer sufficient in an era where models can infer or reconstruct personal data from complex patterns. While privacy-enhancing technologies such as federated learning and differential privacy offer potential solutions, their real-world effectiveness remains uneven and context-dependent.

The review stresses that privacy protections must be evaluated not only in technical terms but in relation to clinical outcomes and patient trust. Systems that technically comply with data protection laws may still erode confidence if patients do not understand how their data are used or if consent mechanisms are unclear. The authors argue that governance frameworks must integrate privacy considerations into procurement, deployment, and monitoring rather than treating them as a one-time compliance hurdle.

Human oversight must be operational, not symbolic

The study also highlights the gap between the concept of human oversight and its implementation in healthcare AI systems. Many AI tools are marketed as decision-support systems that keep clinicians "in the loop." In practice, however, oversight is often poorly defined. Clinicians may lack the training, time, or authority to meaningfully challenge AI recommendations, especially in high-pressure environments.

The authors argue that human oversight must be operationalized through clear decision rights, escalation pathways, and institutional support. Simply requiring a human to review an AI output does not ensure meaningful control if that review is perfunctory or constrained by workflow pressures. Effective oversight requires that clinicians understand the system's limitations, have access to performance information, and possess the authority to override or disable the system when necessary.

Next up, the study highlights training as a critical but often neglected component. Many healthcare professionals receive minimal education on how AI systems function, how they can fail, and how to interpret their outputs critically. Without this knowledge, human oversight risks becoming symbolic rather than substantive. The authors call for governance models that integrate training, competency assessment, and ongoing support as core elements of AI deployment.

The review also addresses procurement as a key leverage point for governance. Healthcare institutions often acquire AI systems through contracts that limit access to performance data or restrict independent evaluation. The authors argue that governance-by-design must begin at procurement, with requirements for transparency, auditability, and post-market monitoring written into contracts. Without such provisions, institutions may find themselves locked into systems they cannot adequately govern.

Toward governance by design in healthcare AI

The study calls for a shift from principle-based ethics to governance by design. This approach embeds fairness, transparency, safety, and oversight into the technical and organizational structures that shape how AI systems are used over time. Rather than treating governance as an external layer applied after deployment, the authors argue that it must be integrated into every stage of the AI lifecycle.

This includes pre-deployment assessment that goes beyond accuracy metrics, deployment strategies that align with clinical workflows, and continuous monitoring that tracks performance across patient groups. It also requires clear allocation of responsibility among developers, healthcare providers, and regulators, supported by documentation and evidence that can withstand scrutiny.

The authors note that emerging frameworks in Europe and elsewhere are moving toward lifecycle-based oversight of AI. However, they caution that regulation alone will not resolve governance failures unless healthcare institutions build internal capacity to manage AI systems effectively. Compliance without operational capability risks producing formal adherence without substantive protection.