AGI boom could strain human oversight and raise systemic risk

The global race to deploy advanced artificial intelligence (AI) has focused largely on speed, scale, and computational power. A new economic framework suggests that the decisive factor shaping the AI transition may be far less visible: the limited human capacity to audit, verify, and underwrite autonomous systems as they expand across industries.

In Some Simple Economics of AGI, published on arXiv, researchers present a model showing that as automation costs fall and agentic systems proliferate, the scarcity of verification capacity becomes the central economic constraint, reshaping incentives for firms, workers, and governments alike.

The measurability gap and the verification bottleneck

As compute power scales and AI systems improve, the cost of automation declines exponentially. At the same time, the cost of human verification remains constrained by biology, time, and embodied experience. This divergence produces a gap between execution and oversight capacity, determining how much AI output can be safely deployed.

The authors describe the current human-in-the-loop equilibrium as dynamically unstable. In the early stages of AI deployment, humans remain central to supervision, auditing, and decision-making. But as firms substitute human labor with cheaper agentic systems, apprenticeship pathways collapse. Entry-level tasks that once trained future experts are automated away. The study calls this erosion the Missing Junior Loop. Without routine practice and exposure to real-world failure modes, the future stock of human expertise shrinks.

This decline in experiential learning undermines long-term oversight capacity. In many domains, theory and formal education alone cannot substitute for hands-on execution. As automation drives human execution time toward zero, the feedback loops that generate tacit knowledge weaken. Over time, this reduces society's ability to audit complex AI systems, increasing the risk of misalignment and systemic error.

The model also highlights a powerful economic distortion. When firms deploy AI systems, they often internalize only a fraction of the social costs associated with failure. If liability regimes are weak or harms are diffuse, the private incentive to invest in verification falls below the socially optimal level. In the extreme case, verification effort can vanish entirely, leading to a surge in unverified throughput.

This underinvestment creates what the authors liken to a lemons market for agentic labor. Buyers struggle to distinguish between well-verified systems and reckless deployments. As cheaper, unaudited alternatives proliferate, high-quality deployment is crowded out. Measured output may rise, but hidden risk accumulates in the background.

The paper further warns against the temptation to use AI to verify AI. While automated oversight tools can expand human bandwidth, correlated blind spots may propagate false confidence. If verification systems share the same training data and structural biases as the agents they audit, systemic errors can scale rapidly.

From automation to underwriting: A structural shift in competition

The authors note that competitive advantage migrates away from raw automation and toward verification-grade infrastructure. In an economy where measurable execution becomes commoditized, scarcity shifts to ground truth, liability underwriting, and the ability to certify outcomes.

The study proposes that governments and institutions must actively force risk internalization. Mechanisms such as strict liability regimes, mandatory insurance, audit trail requirements, and cryptographic provenance standards serve to price tail risk and create demand for verification. By making failure costly, these interventions align private incentives with social welfare.

When liability is priced, the competitive landscape transforms. Agentic software companies stop competing solely on automation scale and begin competing on underwriting capacity. Firms that can absorb downside risk and provide verifiable guarantees gain an edge. The product shifts from standalone AI agents to insured outcomes.

The paper suggests that this dynamic may lead to vertical integration or tight partnerships between AI platforms and insurers. Firms controlling execution environments can observe risk formation in real time and price it dynamically. In such a system, verification logs, claims data, and model improvement loops become strategic assets.

Ground truth emerges as a durable moat. Data that lowers the cost of verification, rather than merely the cost of execution, becomes especially valuable. Execution-grade data teaches systems what to do, but verification-grade data determines whether outcomes can be trusted and insured. As general-purpose models commoditize execution, companies with access to scarce, high-fidelity ground truth maintain defensible positions.

The framework extends beyond firms to investors and individuals. For investors, value shifts toward funding what is not yet measurable but can be made verifiable. Deep technology sectors where physical validation and real-world testing remain essential may see accelerated investment as AI reduces software scaffolding costs.

For individuals, the model predicts a fundamental change in the nature of work. As intelligence becomes abundant, roles centered on routine execution decline in value. Human labor migrates toward intent direction, oversight, underwriting, and creative activities rooted in social connection and coordination.

The risk of a hollow economy and the path to an augmented one

If left unmanaged, the forces described in the paper exert a gravitational pull toward what the authors term a Hollow Economy . In this scenario, nominal output expands rapidly as automation scales. However, human agency decays. Verification bandwidth fails to keep pace with execution power. Hidden risk accumulates as misaligned or unaccountable outputs propagate through production systems.

This outcome is not inevitable. By scaling verification capacity alongside agentic capability, societies can transform cognitive acceleration into an engine of discovery and experimentation. The decisive variable is institutional design.

Some economists argue that aggregate productivity gains from AI may remain modest because many high-value tasks resist automation. The authors incorporate this insight but maintain that even if automation drives explosive output in measurable domains, verification remains a strict complement. If even a small share of tasks requires human oversight, that bottleneck constrains overall system performance.

Moreover, under standard assumptions of diminishing marginal utility, the finite upside of rapid growth cannot offset even a small probability of catastrophic misalignment. This implies extreme risk aversion toward unverified deployment unless AI significantly reduces existential threats such as mortality.

The geopolitical dimension further complicates the picture. In strategic competition between nations, relative capability may be valued over absolute safety. This can produce a Prisoner's Dilemma dynamic in which countries underinvest in verification to avoid falling behind rivals. Without coordination, a race to the bottom in oversight standards becomes possible.