Fully automated AI triage faces legal barriers under GDPR and AI Act

Artificial intelligence may improve the speed and accuracy of emergency triage, but in Europe, regulatory scrutiny is intensifying just as quickly as innovation. Algorithms capable of predicting disease severity and classifying patient urgency are moving from research labs into hospital workflows. However, the legal framework surrounding these systems is far stricter than many healthcare providers may anticipate.

In Regulating AI-Driven Triage: Fundamental Rights and Compliance Challenges in the European Union, published in the journal AI, researchers examine how EU law constrains the deployment of AI in emergency settings. The study finds that under the AI Act and the GDPR, fully automated triage decisions are difficult to justify, reinforcing the requirement for meaningful human intervention.

AI's growing role in emergency triage

Emergency triage is designed to prioritize patients when demand exceeds available medical resources. Traditionally performed by trained clinicians using standardized tools, triage now increasingly incorporates machine learning models capable of processing large volumes of patient data within seconds.

The study outlines how AI systems have demonstrated strong predictive performance in emergency settings. Traditional machine learning tools have shown high discriminatory capacity in identifying high-risk patients, predicting hospital admissions, and stratifying disease severity. Deep learning and ensemble models frequently outperform conventional triage scoring systems. In trauma care and acute settings, AI systems have improved predictions of mortality risk, need for intensive care, and likelihood of hospitalization.

Generative AI tools based on large language models are also entering the triage landscape. These systems can process unstructured clinical notes, interpret symptom descriptions, and assist with urgency classification. Reported accuracy rates for large language models in structured triage-like scenarios range between roughly 70 percent and 90 percent, often approaching clinician agreement under controlled conditions. They can assist in prioritizing emergency calls, supporting prehospital assessments, and enhancing training in emergency departments.

Despite these performance gains, the authors warn that technical accuracy does not automatically translate into legal or ethical acceptability. Triage decisions carry significant consequences, affecting access to care, treatment timing, and potentially survival outcomes. As a result, AI deployment in this context triggers strict regulatory scrutiny under EU law.

High-risk AI under the EU AI Act

The study places particular focus on the EU AI Act, which classifies emergency healthcare triage systems as high-risk AI systems. This classification activates a set of stringent obligations for developers and deployers.

Under the AI Act, high-risk systems must be designed to allow effective human oversight. This requirement goes beyond symbolic supervision. Systems must enable trained professionals to understand outputs, detect anomalies, interpret recommendations correctly, and override decisions when necessary. Oversight must be proportionate to the system's autonomy and risk profile.

The authors stress that the emergency setting complicates this requirement. Triage unfolds under time pressure, often with limited staff and unpredictable patient surges. Full verification of every AI-generated recommendation is not operationally feasible. Instead, the study suggests that oversight must be layered and organizational rather than purely individual. Real-time clinical judgment should be combined with escalation mechanisms for uncertain or high-risk cases, structured override capabilities, post-incident audits, and continuous monitoring.

The AI Act also mandates that deployers conduct a Fundamental Rights Impact Assessment before using high-risk AI systems. This assessment requires mapping potential impacts on rights such as human dignity, equality, data protection, access to healthcare, and effective remedy. Deployers must evaluate foreseeable misuse, automation bias, and the severity and likelihood of potential harms. Risk mitigation strategies must be documented and integrated into governance processes.

The study notes that while the AI Act requires these assessments, practical implementation guidance remains limited. Institutions will need structured internal procedures to ensure that triage systems are validated, monitored, and reviewed throughout their lifecycle.

Transparency obligations further shape AI use in triage. Individuals subject to decisions based on high-risk AI systems are entitled to a clear and meaningful explanation of the system's role and the main elements influencing the decision. Patients must also be informed when AI is used in decision-making. In emergency contexts, delivering explanations in real time may be challenging, but the study argues that ex post transparency remains essential to preserve trust and accountability.

GDPR and the limits of fully automated decisions

GDPR imposes additional constraints. AI-driven triage typically involves profiling because personal health data are processed to evaluate and predict aspects of an individual's medical condition.

When profiling produces legal or similarly significant effects, Article 22 of the GDPR restricts decisions based solely on automated processing. The authors argue that most AI-driven triage systems would fall within this category because they significantly affect patients' access to healthcare and treatment priority.

Under GDPR standards, fully automated triage decisions are difficult to justify. The regulation favors the inclusion of meaningful human intervention. Human review must be substantive, not a routine endorsement of algorithmic output. The person reviewing must be authorized, competent, and capable of modifying the decision. Automation bias, where clinicians defer excessively to algorithmic outputs, presents a significant risk and must be addressed through training and workflow design.

Although the GDPR allows limited exceptions to the prohibition on fully automated decisions, including explicit consent or authorization by law, the study considers these paths problematic in triage settings. Patients in emergency conditions may lack the capacity to provide valid consent, and the irreversible consequences of triage decisions make purely automated models ethically and legally fragile.

Overall, the EU data protection law strongly favors hybrid models in which AI supports clinical decision-making but does not replace human judgment.

Medical devices regulation and general-purpose AI

The Medical Devices Regulation adds another layer of regulatory complexity. If an AI system directly determines clinical priority, suggests specific medical actions, or influences access to treatment, it may qualify as a medical device. In that case, manufacturers must meet safety, performance, documentation, and risk management obligations under the MDR.

Standalone AI tools used specifically for medical purposes may fall within this regime. However, general-purpose AI models, such as widely available large language models, occupy a gray area. If used informally by clinicians without being intended or marketed for medical use, they may not automatically qualify as medical devices. When integrated into structured triage systems with clinical intent, however, they could trigger concurrent compliance with both the AI Act and the MDR.

Responsibility allocation becomes critical. Developers are accountable for system design, training data quality, risk mitigation, and compliance with AI Act obligations. Deployers, including hospitals and public authorities, bear primary responsibility for integrating AI into clinical workflows, conducting impact assessments, ensuring oversight, and meeting data protection requirements. Clinicians retain professional responsibility for patient care but should not carry liability for systemic design flaws or institutional constraints.

The study notes that regulatory compliance does not replace existing medical liability regimes. Instead, AI governance operates alongside professional accountability and product liability frameworks.