The Four Layers of AI Failure

 

When AI systems fail, many people instinctively reach for the hackneyed expression “AI hallucination.”

It is a convenient label, sure, but it is hardly diagnostic.

A context-tracking failure, a reasoning failure, or a verification failure may all be labeled as “AI hallucination.” But knowing that the output is wrong is not the same as knowing the actual source of the problem.

So when AI’s response looks like a “hallucination,” it helps to ask:

“At which layer did the failure occur?”

Because poor prompts cannot be necessarily fixed by better harness architecture, and prompt engineering does not necessarily solve verification problems.

A better approach to fixing AI failures requires separating the mechanics into three distinct levels of analysis:

• Internal layers during a single token generation
• Autoregressive trajectory formation
• External orchestration

Each layer operates differently. Each fails for different reasons. And collapsing them into a single narrative leads to mistaken conclusions.

Internal Layers During a Single Token Generation

At the smallest scale, LLMs perform a sequence of mathematical transformations to generate a probability distribution over the next token.

At this level, the model is not constructing a narrative in the way we typically imagine people constructing thoughts. It is transforming one high-dimensional context state into another, until a next-token distribution emerges.

Failure modes at this level are scale-dependent in a very specific sense: small changes to the input can produce disproportionately large changes to the internal transformations.

This happens because every input token contributes to the model’s internal representation of context.

When you compose a prompt, you are not merely adding factual information. Your choice of words and symbols, as well as the exact way you arrange them, determines which learned features become active in the internal layers.

Two prompts that share nearly identical meaning from a human perspective may traverse entirely different regions of representation space inside the model, leading to qualitatively different outputs.

There is a growing body of evidence supporting this, ranging from the experiments demonstrating that negative prompts often reduce factual accuracy and amplify bias to the studies demonstrating that human-perceived similarity in prompts does not cleanly map to similarity in LLMs’ internal representations of these prompts.

One concept that becomes extremely relevant at this level of analysis is “attention heads.”

An attention head can be understood as a mechanism that selectively determines which parts of the existing context should influence the current token-generation step.

Some heads appear to track local grammatical dependencies. Others attend to quotation boundaries, code structure, reasoning chains, or various semantic relationships.

It should be noted that attention heads cannot be cleanly separated and are not fully understood. The important take-away, however, is that a seemingly insignificant change in prompt composition can cause different attention patterns to emerge.

And depending on which attention heads become active, the model may retrieve different contextual features and ultimately arrive at a different next-token distribution.

For example, adding a phrase intended to increase caution may inadvertently shift attention toward patterns associated with uncertainty, refusal behavior, or speculative language.

While attention heads determine what information becomes salient at a given step, the eventual output depends on how that information is transformed and combined throughout the rest of the generation. Even if every individual token is generated through locally coherent computations, the growing sequence of tokens may still produce undesirable model behavior over time.

Understanding how this happens requires moving beyond single-token generation and looking at autoregressive trajectory formation.

The Trap of Self-Conditioning

Because large language models generate text auto-regressively, every newly produced token immediately becomes part of the context used to generate the next one.

The model is therefore “taking into account” not only the user’s prompt but also its own previously output tokens in the growing response sequence.

Suppose the model begins constructing an argument in favor of perspective A. The first few sentences establish concepts, terminology, and assumptions associated with that perspective.

This doesn’t mean that the model adopts perspective A the way a human would. Rather, perspective A becomes embedded in the context against which every future token is evaluated.

Once a growing sequence accumulates sufficient structure, reversing course becomes progressively less likely. As a result, errors introduced early in a generation may continue to shape subsequent predictions and can compound over time.

This is also one of the reasons iterative prompting often works well. Each revision changes the conditioning context, allowing the model to begin a new trajectory instead of remaining anchored to an earlier one.

Many users have also noticed that regenerating a response to the same prompt can produce dramatically different results. This happens because regenerating a response means sampling different early tokens, which can significantly alter the overall generative trajectory.

In addition, autoregressive self-conditioning can help explain why intermediate reasoning traces can improve or degrade the model’s performance.

When a model is trained to generate extensive hidden reasoning before its final answer, it gains additional autoregressive steps that may allow it to explore alternative “perspectives.”​

Mechanistically, each reasoning token reshapes the generation trajectory, becoming part of the context that influences the next reasoning token, allowing the model to progressively steer its own “thinking” before producing the final output.

So cranking the model’s thinking effort up is best understood as increasing inference-time computation through additional autoregressive self-conditioning, rather than making the model “think harder” in a human sense.

It’s worthwhile noting that additional reasoning tokens are not guaranteed to improve performance. If the reasoning begins from flawed assumptions, longer reasoning can sometimes reinforce those assumptions rather than correct them.

In one of my earlier articles, I analyzed a study that examined Chain-of-Thought prompting for various tasks and demonstrated that it is not a neutral transparency tool. Sometimes CoT corrects reasoning, sometimes it amplifies steering, and sometimes it actually reduces accuracy.

This is described as a snowball error effect and even structure snowballing in more recent LLM research.

Because the model conditions on its own previous output, correcting a bad trajectory is often easier by restarting the reasoning from a revised prompt, which brings us to the next level of analysis — the harness.

The System Level Failures

The preceding failure modes originate within the model itself. However, modern AI systems increasingly fail — or succeed! — for reasons that lie outside the neural net.

A language model can predict what a correct SQL query looks like, but only an external action can determine whether the query returns the intended records.

Therefore, it is important to understand that prediction and verification are separate computational processes. Performing probabilistic inference over token sequences isn’t enough if the task requires checking whether the prediction is actually true within a particular environment. Verification often requires deterministic operations over grounded states of the world.

Many failures labeled “hallucinations” are actually more accurately described as verification failures. Increasing model size, extending context length, or improving prompting cannot substitute for deterministic operations over grounded states.

This explains why modern AI engineering increasingly focuses on architectures rather than models in isolation. The focus moves away from prompt engineering to figuring out which additional components should verify, execute, retrieve, or monitor the model’s outputs.

The Practical Rule

Before trying to improve AI’s performance, identify which layer you should be looking at.

Is the task purely inferential?

If the task only requires synthesizing knowledge already contained in the model (e.g., explaining a concept or summarizing a paper), better prompting or additional reasoning may improve performance.

Does the task execution depend on external state?

If correctness depends on a database, API, filesystem, current events, or physical measurements, no amount of prompting can guarantee correctness without external verification. If an objective verifier exists, you shouldn’t rely solely on the model’s confidence.

A useful one-liner I heard somewhere:

“Use the model to propose and the environment to dispose.”

The Ultimate Layer: Human Goals

There is one final layer that sits above the others: human goals.

Everything below this level deals with whether a system computes correctly. Everything at this level deals with whether the computation was worth performing in the first place.

A language model may produce a fluent response. A verification system may run deterministic verifications. External tools may execute flawlessly. Yet the system can still fail if it does not advance the objective it was built for.

At the human level, we ultimately care if the system accomplishes our goals. And this is something no model, database, or verification layer can determine on their own. Better internal representations, better predictions, and better verifications do not automatically produce better human outcomes.

As Peter Drucker so elegantly noticed,

“There is nothing so useless as doing efficiently that which should not be done at all.”