Hidden risks in classroom AI: Bias, errors, and opaque systems

A new study has found that widely used generative artificial intelligence tools designed for education are failing to meet basic transparency standards, raising concerns about trust, accountability, and safe classroom adoption. The research highlights critical gaps in how these systems disclose their inner workings, data use, and limitations.

Published in AI & Society, the study titled "Hard to find, harder to understand: examining transparency in educational generative AI" systematically evaluates 20 AI tools used by teachers and finds widespread inconsistency and opacity in the information provided to users.

AI tools for teachers offer functionality but withhold critical system details

Conducted by Mazid Ul Hasan of the University of Cincinnati, the study reveals that most educational AI tools emphasize ease of use and productivity while providing little to no explanation of how their systems actually generate content. Out of the 20 tools analyzed, 16 failed to explain basic mechanisms of generative AI, such as how models process prompts or produce outputs based on training data.

Instead of offering technical clarity, many platforms focused on user-facing features like lesson generation, quizzes, and feedback tools, leaving educators unaware of what happens between input and output. This absence of explanation limits teachers' ability to evaluate the reliability of AI-generated material and undermines informed decision-making in classrooms.

Only four tools provided any meaningful description of how generative AI works, and even among these, the depth and clarity varied significantly. Some tools briefly noted that AI models learn from large datasets, but stopped short of explaining how those patterns translate into specific outputs. One tool stood out by detailing a step-by-step process of how text is analyzed, keywords extracted, and questions generated, showing that meaningful transparency is achievable.

The study also examined disclosures about underlying AI models, such as large language models. While 18 tools referenced the use of such models, the information was often incomplete or difficult to locate. Nine tools indicated reliance on OpenAI's GPT models, while others mentioned using multiple models without naming them. In several cases, this information was buried in privacy policies or indirect references, making it inaccessible to typical users.

This lack of clarity reinforces concerns that educators are interacting with systems they do not fully understand, despite relying on them for instructional decisions.

Missing data transparency and limited disclosure of AI training raise ethical risks

The near-total absence of transparency regarding training data and content sources. None of the analyzed tools disclosed specific datasets or explained how their models were trained, leaving critical questions about bias, accuracy, and intellectual property unanswered.

Generative AI systems rely heavily on large datasets, often scraped from public or proprietary sources. Without disclosure of these sources, it becomes impossible to determine whether outputs are influenced by biased, outdated, or low-quality information. This is particularly concerning in educational contexts, where accuracy and fairness are essential.

The study also highlights a lack of attribution in AI-generated content. Only one tool provided references or links to the sources of its generated material, while the rest offered outputs without any indication of origin. This raises concerns about academic integrity and copyright compliance, as educators and students are unable to trace or verify the information provided.

Privacy transparency presents a more mixed picture. A majority of tools stated that user data is not used to train AI models, reflecting growing regulatory pressure around data protection. However, several tools failed to address this issue altogether, and one indicated that user interactions may contribute to model training.

Even when privacy disclosures were present, they often lacked clarity or were embedded in complex legal language. This creates a situation where users may assume safety without fully understanding how their data is handled.

Transparency in one area, such as privacy, does not compensate for opacity in others, such as training data or system design. Partial transparency can lead to overconfidence in AI tools, increasing the risk of misuse.

Limited warnings about AI errors leave educators vulnerable to misuse

The study finds that most educational AI tools fail to adequately communicate their limitations. Only seven of the 20 tools provided clear warnings about potential inaccuracies, biases, or errors in generated content.

These disclaimers, when present, were often placed directly within user interfaces, encouraging educators to verify outputs and apply professional judgment. Some tools suggested practical approaches, such as reviewing AI-generated material before classroom use, which aligns with responsible AI practices.

However, the majority of tools either omitted such warnings entirely or confined them to terms and conditions pages, where they are unlikely to be seen. In these cases, statements about inaccuracies were framed as legal disclaimers rather than practical guidance, offering little value to educators.

The absence of visible and actionable warnings increases the risk that teachers may rely on flawed or biased content without verification. This is particularly concerning in scenarios where AI-generated material could reinforce stereotypes, propagate misinformation, or misrepresent subject matter.

The study also points to the role of design in shaping transparency. Information about AI systems was often scattered across websites, hidden in policy documents, or written in technical or legal language that is difficult for educators to interpret. Even when relevant details were available, poor placement and unclear wording reduced their usefulness.

On the other hand, tools that presented information in clear, accessible formats, such as FAQs or dedicated explanation pages, demonstrated that effective transparency is not only possible but practical.

Transparency gaps threaten trust and responsible AI adoption in education

The current state of transparency in educational generative AI tools is insufficient to support responsible use. While these systems offer significant benefits in terms of efficiency and innovation, their lack of openness about key aspects undermines trust and accountability.

Transparency is closely linked to user trust, particularly in high-stakes environments like education. When educators understand how AI systems work, their limitations, and the risks involved, they are better equipped to use them effectively. Conversely, a lack of transparency can lead to both overreliance and distrust, each carrying its own risks.

The research highlights the need for a balanced approach, where educators neither blindly trust nor entirely reject AI tools. Achieving this balance requires clear, accessible information about how systems operate, what data they use, and where they may fail.

To address these challenges, the study calls for stronger commitments from developers, policymakers, and educators. Developers are urged to provide centralized, user-friendly explanations of their systems, disclose training data practices, and clearly communicate limitations. Policymakers are encouraged to enforce transparency requirements through regulations and procurement standards.

Educators, meanwhile, are advised to critically evaluate AI tools before integrating them into teaching practices. This includes examining available information about models, data use, and limitations, and avoiding tools that fail to provide adequate transparency.

The study also points to a growing need for training and professional development, enabling teachers to understand and assess AI systems more effectively.