LLM observability requires specialized monitoring for non-deterministic outputs

Observability for Large Language Models (LLMs) requires different metrics than traditional application performance monitoring (APM) due to the non-deterministic nature of model outputs. While standard APM focuses on request counts, error rates, and latency, LLM observability must track token usage, cost per request, and quality regressions such as hallucinations or data leaks.

In Retrieval-Augmented Generation (RAG) systems, standard model-centric traces often fail to identify why an answer might be incorrect even when technical indicators like latency and status codes appear normal. Effective observability must extend beyond the model call to include the complete evidence path, tracing how queries are rewritten, how sources are fetched, and how information is reranked and selected for the context window. This allows developers to distinguish between generation failures and retrieval failures, such as using stale or irrelevant data.