Article
Detecting AI Agent Failure Modes in Production - Latitude.so
latitude.so
Quoted on this wiki
Every place a page here uses this source, in the order the words come in it.
The agent gradually shifts from the user’s original objective over the course of a long workflow. A user asks the agent to “schedule a meeting with the team next week avoiding Friday.” By step 8, the agent is scheduling for the following month because it over-weighted a scheduling conflict mentioned at step 4 and reinterpreted the original constraint. “According to research on LLM agent benchmarks, agents evaluated only on final-output quality pass 20–40% more test cases than full trajectory evaluation reveals (Wei et al., 2023).” 4. Infinite Loops and Reasoning Stalls
Non-Deterministic Execution Paths “You can’t evaluate a multi-turn agent by testing one path — you need to test across the distribution of paths the agent actually takes in production, which requires production trace data, not just synthetic test cases.” State Management Across Steps
How do I detect goal drift in a multi-turn agent? “Set up LLM-as-judge evaluation that receives the full conversation history (including turn 1) and scores whether the final output addresses the user’s original objectives.” Why don’t standard monitoring tools work for AI agents?
How should I prioritize which failure modes to instrument first? “Then add continuous quality evaluation to catch goal drift and context loss.” Related: Multi-turn conversation tracing in Latitude · Auto-generated evals with GEPA · Latitude Evals product page