AI evaluation
Agent Evaluation Framework
Standardized evaluation of AI agents across benchmarks and quality metrics.
PythonPostgreSQLGrafanaRaypytest
- Challenge
- Agent quality (correctness, groundedness, hallucination) needs repeatable measurement before production rollout.
- My Role
- Built the framework: benchmark/task loaders, 10 reusable metrics (correctness, groundedness, hallucination, citations, navigation, guardrails, latency, tokens, LLM-judge, completeness), Postgres result storage, Grafana dashboards, Ray parallel execution, 150+ tests.
- Architecture
- Core loaders + metric registry feed agent solution functions; results persist to Postgres and visualize in Grafana.
- Interesting decisions
- Scores 0.0–1.0 per metric with explicit pass thresholds; duplicate-aware result export; evaluation configs live with the agent under test.
- Result
- Reusable harness used for OA Help Agent and Data Science Agent comparisons.
- What I Learned
- Grade what the agent retrieved, not just what it answered — retrieval visibility catches grounding failures.