Medium LLM
The Four Layers of AI Failure
eval
What happened
The article argues that 'hallucination' is an overused, non-diagnostic term. It proposes analyzing AI failures across three distinct layers: internal token generation transformations, autoregressive trajectory formation, and external orchestration. Each layer requires different debugging and mitigation strategies.
Why it matters
It provides a structured taxonomy for debugging LLM failures, helping builders target the correct layer (prompt, model, or orchestration) for fixes.
The take
Breaking down failures into these layers is highly practical for system designers. Too many teams try to fix orchestration-level failures with prompt engineering, or vice versa. Understanding where the failure occurs is key to building robust evals.
Do this
Use this three-layer taxonomy (token, trajectory, orchestration) to categorize and debug failures in your LLM evaluation pipelines.
Don't read this site daily. Get it in your inbox.
The daily brief and Sunday deep dive — distilled, scored, and opinionated. For builders only.