Traditional monitoring tells you a service is up. An AI system can be up, fast and confidently wrong. Observability for AI has to cover quality as well as health.
The signals that matter
- Traces: the full chain for each request: input, retrieved sources, tool calls, output, latency and token use.
- Quality: scores against an evaluation set built from your own data, re-run on every model, prompt or data change.
- Grounding: whether answers cite sources, and how often the system correctly says "I don't know".
- Cost: spend per request and per day, with limits and alerts before the invoice arrives.
- Feedback: what reviewers approve, edit or reject, fed back into the evaluation set.
LLM-as-a-judge, calibrated
An LLM judge can score far more output than a person can review. It is only trustworthy once it agrees with your reviewers on a sample they have scored. Calibrate first, then let it run, and keep sampling.
Tooling
LLM observability platforms such as Langfuse (not to be confused with the LangChain framework) handle tracing and evaluation runs; your cloud provider's logging covers the infrastructure. What matters more than the tool is deciding, before go-live, which numbers would make you roll back.