Integrity hallucination raises concerns over inconsistent AI decision-making in high-stakes systems

A new working paper by Seulki Lee of the AI Integrity Organization is calling for a fundamental shift in how artificial intelligence (AI) systems are governed, arguing that current frameworks fail to examine the most critical component of AI decision-making: the reasoning process itself. It warns that evaluating outputs alone is no longer sufficient to ensure trust, accountability, or fairness.

The study, titled "AI Integrity: A New Paradigm for Verifiable AI Governance," published as an AIO Working Paper, introduces a new governance model centered on verifying how AI systems arrive at decisions rather than simply assessing the results they produce. The research positions "AI Integrity" as a fourth paradigm alongside existing frameworks of AI Ethics, AI Safety, and AI Alignment, each of which, it argues, leaves a critical gap in process-level accountability.

Why existing AI governance models fail to verify reasoning

The study identifies a structural limitation shared by the three dominant AI governance paradigms. AI Ethics focuses on whether outcomes are morally acceptable, AI Safety addresses system robustness and protection from harm, and AI Alignment evaluates whether systems behave according to human preferences. While each framework serves a distinct purpose, all rely on outcome-based evaluation.

According to the research, this approach creates a blind spot. An AI system may produce acceptable or even optimal outcomes while relying on inconsistent value hierarchies, biased evidence selection, or selective use of data. These underlying issues remain invisible when only outputs are assessed.

The paper highlights that current frameworks assume that if the outcome meets predefined criteria, the reasoning behind it must also be sound. However, this assumption breaks down in complex systems where decisions are influenced by multiple layers of data, sources, and value judgments. A system could reach the correct conclusion through flawed reasoning, or conversely, produce harmful outcomes despite following internally consistent logic.

This gap becomes particularly concerning in high-stakes applications. In healthcare, for example, an AI system recommending treatment may rely on incomplete or selectively weighted evidence. In financial systems, loan approvals may reflect hidden biases embedded in data selection or source prioritization. In legal contexts, decision-support tools may apply inconsistent standards without any mechanism to verify their internal reasoning.

The study argues that without a framework to examine how decisions are made, governance efforts remain incomplete. This limitation has become more pronounced as AI systems grow more complex, making it increasingly difficult to trace their internal processes using traditional methods.

The Authority Stack: A new model for understanding AI reasoning

The study introduces "Authority Stack," a four-layer model designed to deconstruct how AI systems make decisions. This framework provides a structured way to analyze the reasoning process by separating it into distinct but interconnected components.

• The top layer, known as normative authority, defines the values that guide decision-making. These values determine how trade-offs are handled when competing priorities arise. For example, a system may prioritize safety over efficiency or fairness over profitability.
• The second layer, epistemic authority, governs what types of evidence are considered valid. This includes the standards used to evaluate information, such as whether the system prioritizes scientific studies, expert opinions, or anecdotal data.
• The third layer, source authority, determines which sources are trusted. This involves assessing the credibility of institutions, individuals, or datasets that provide information. Different systems may assign varying levels of trust to sources based on factors such as expertise, reliability, or relevance.
• The fourth layer, data authority, represents the actual data selected for decision-making. This layer is influenced by the three above it, as values, evidence standards, and source preferences collectively determine what information is included or excluded.

The study states that these layers operate as a cascade, with higher levels influencing those below. A system's value framework shapes how it evaluates evidence, which in turn affects the sources it trusts and the data it ultimately uses. This cascading structure provides a comprehensive view of how decisions are constructed.

Notably, the research distinguishes between legitimate interactions across layers and what it terms "Authority Pollution." While it is natural for values to influence decision-making, problems arise when this influence becomes opaque, inconsistent, or distorts factual accuracy. For instance, prioritizing inclusivity at the value level may lead to the suppression or alteration of factual data if not properly managed.

By identifying these dynamics, the Authority Stack offers a way to diagnose and address hidden biases within AI systems. It shifts the focus from isolated outputs to the full chain of reasoning, enabling a more thorough evaluation of system behavior.

Integrity hallucination and the rise of inconsistent AI reasoning

The study also introduces a new risk concept known as "Integrity Hallucination," describing situations where AI systems generate inconsistent or unstable value judgments across similar scenarios. This phenomenon reflects a lack of coherent internal reasoning, even when outputs appear plausible.

Integrity Hallucination can manifest in several ways. In some cases, systems exhibit random variation, producing different answers to identical questions due to stochastic processes. In others, responses change based on subtle differences in phrasing or context, indicating sensitivity to framing rather than adherence to consistent principles.

The most severe form occurs when no stable value structure exists at all. In such cases, the system's responses are driven by pattern recognition rather than principled reasoning, making behavior unpredictable and difficult to audit. This raises significant concerns for high-stakes applications where consistency and reliability are essential.

The research highlights that Integrity Hallucination is not merely a theoretical issue but an observable phenomenon across multiple AI models. Variations in consistency rates suggest that even advanced systems may lack stable reasoning frameworks, reinforcing the need for process-level verification.

To address this challenge, the study proposes a set of measurement tools within the PRISM framework. These tools assess factors such as consistency across repeated scenarios, sensitivity to contextual changes, and alignment between different layers of the Authority Stack. By quantifying these aspects, the framework aims to transform abstract concerns about AI reasoning into measurable metrics.

From outcome evaluation to process verification in AI governance

The introduction of AI Integrity represents a shift from evaluating what AI systems do to understanding how they do it. This change has significant implications for policymakers, developers, and organizations deploying AI technologies.

Under the proposed framework, accountability is no longer limited to ensuring that outputs meet ethical or safety standards. Instead, systems must demonstrate that their reasoning processes are transparent, consistent, and auditable. This requires new tools, methodologies, and regulatory approaches capable of examining internal decision structures.

The study's PRISM framework provides a roadmap for implementing this approach. It outlines a multi-phase research program designed to measure each layer of the Authority Stack independently and assess their interactions. By doing so, it aims to create a comprehensive profile of an AI system's reasoning behavior.

One of the key goals of this framework is predictive capability. By understanding how a system processes values, evidence, and sources, it becomes possible to anticipate its responses to new scenarios. This could enable pre-deployment testing and risk assessment, reducing the likelihood of unexpected or harmful outcomes.

AI Integrity is a procedural concept rather than a normative one. It does not prescribe which values are correct or which decisions should be made. Instead, it requires that whatever values a system holds are applied consistently and transparently. This distinction allows for flexibility across different cultural, institutional, and regulatory contexts.

However, the study acknowledges that AI Integrity alone is not sufficient to ensure trustworthy AI. A system could maintain perfect internal consistency while operating on harmful or biased values. As a result, AI Integrity is positioned as a necessary complement to existing frameworks rather than a replacement.

A new direction for AI accountability

The findings suggest that the future of AI governance may depend on the ability to move beyond surface-level evaluation and toward deeper analysis of reasoning processes. As AI systems become more integrated into critical decision-making, the need for verifiable and auditable reasoning will only grow.

The research provides a new lens through which to evaluate these systems. It highlights the importance of examining not just outcomes, but the full chain of decisions that lead to them. This approach has the potential to uncover hidden biases, improve transparency, and strengthen trust in AI technologies.

The study also raises important questions about how such frameworks can be implemented in practice. Measuring reasoning processes in complex systems remains a significant technical challenge, and the proposed methodologies will require further validation and refinement.