Blind trust in AI models could lead to flawed decisions across industries

A new study by Justin Grandinetti of the University of North Carolina at Charlotte challenges one of the most dominant narratives in artificial intelligence: that modern AI systems are inherently unknowable "black boxes." The research argues that this perception, reinforced by both industry and academic discourse, is obscuring critical understanding of how widely used machine learning models actually function, and how they can fail.

Published in AI & Society, the study titled "Beyond black boxes and the AI sublime: critically assessing the code behind commonly used machine learning models" examines AI at the level of code, combining technical analysis with sociotechnical critique.

Breaking the 'Black Box' narrative in AI systems

The study identifies two dominant and mutually reinforcing narratives shaping public and academic understanding of AI: the "black box" and the "technological sublime." Together, these ideas portray AI as both opaque and almost mystical, limiting meaningful scrutiny of its inner workings and societal consequences.

The black box metaphor suggests that while inputs and outputs are visible, the internal processes of AI systems remain hidden and inaccessible. This framing has deep historical roots and is now widely used to describe machine learning models whose decision-making processes are difficult to interpret. However, the research argues that this narrative can lead to misplaced trust in systems that may contain errors, biases, or flawed assumptions.

The concept of the technological sublime positions AI as an inevitable and powerful force, capable of transforming society in ways that are beyond human control. This perception discourages critical engagement, reinforcing the idea that AI systems should be accepted rather than questioned.

The study warns that these narratives together create what scholars describe as an "algorithmic drama," where AI is seen as both highly influential and fundamentally unknowable. This dual framing not only distorts public understanding but also limits academic inquiry, particularly in the humanities and social sciences, where technical aspects of AI are often overlooked.

Instead of accepting opacity as inevitable, the research calls for a shift toward examining the actual code and architecture of AI models. It argues that understanding even basic computational structures can help demystify AI and reveal how decisions are produced.

How machine learning models actually work and where they fail

To move beyond abstract debates, the study analyzes two widely used machine learning models: linear regression and decision trees. These models are commonly applied across industries due to their relative simplicity, efficiency, and interpretability compared to more complex deep learning systems.

Linear regression, one of the most fundamental machine learning techniques, is used to identify relationships between variables and make predictions based on those relationships. It is widely applied in fields such as business forecasting, healthcare research, and agriculture. However, the study highlights significant limitations that can lead to misleading conclusions.

One key issue is the confusion between correlation and causation. The research illustrates this with a well-known example: a strong statistical relationship between ice cream sales and murder rates. While the data may show a correlation, the underlying cause is an external variable, such as temperature, which influences both factors. Without accounting for such variables, linear regression models can produce conclusions that appear valid but are fundamentally flawed.

The model is also highly sensitive to outliers and missing data, which can distort results. In real-world applications, this can lead to incorrect predictions or policy decisions, particularly when models are used without proper contextual understanding.

Decision trees, another commonly used model, operate differently by breaking down data into a series of decision points, similar to a flowchart. These models are widely used in credit risk assessment, customer analytics, and classification tasks. They are valued for their ability to handle both numerical and categorical data and for their visual interpretability.

However, the study finds that decision trees are not immune to bias or error. They can become overly complex when trained on large datasets, a problem known as overfitting, where the model captures noise rather than meaningful patterns. This can result in unstable predictions, where small changes in data lead to significantly different outcomes.

In financial contexts, such as loan approvals, these limitations can have serious consequences. The research notes that models may produce false positives or false negatives, denying credit to eligible applicants or approving risky borrowers. These errors are often accepted as part of the system's trade-offs, raising ethical concerns about fairness and accountability.

According to the study, bias in AI does not arise solely from flawed data. It can also emerge from the choice of model, the design of algorithms, and the assumptions built into the system. This highlights the need for interdisciplinary collaboration to identify and mitigate potential risks.

Why transparency alone cannot solve AI's accountability problem

While calls for greater transparency in AI systems have gained momentum, the study argues that simply making code accessible is not enough to address the challenges posed by machine learning. One major issue is that even when code is available, it may be difficult for non-experts to understand. Machine learning models often operate in ways that are not easily interpretable, even for trained data scientists. This creates a paradox where increased transparency does not necessarily lead to greater accountability or trust.

The research also points to the growing practice of "openwashing," where companies claim to provide open AI systems but restrict access through APIs or licensing agreements. This limits meaningful scrutiny while maintaining the appearance of transparency.

The study advocates for a broader sociotechnical approach that considers how AI systems are developed, deployed, and integrated into society. This includes examining data sources, institutional practices, and power dynamics that shape AI outcomes.

The findings suggest that AI should not be viewed as a standalone technological artifact but as part of a complex system involving human decision-making, cultural norms, and organizational structures. Understanding this interplay is essential for addressing issues such as bias, discrimination, and unintended consequences.

Importantly, the study rejects the idea that only technical experts should engage with AI. It argues that researchers from diverse disciplines, including the humanities and social sciences, must develop a basic understanding of how AI systems work in order to contribute to critical discussions about governance and ethics.