Agents: Add saved test cases with an LLM judge and rerun them before publishing
Fin offers Simulations where one LLM plays the customer and another judges handoff timing, content used and standards, with a saved regression library rerun after every change (https://fin.ai/testing); Lindy has offline evals with LLM-as-judge scorers over reference tasks (https://docs.lindy.ai/fundamentals/lindy-101/evals); OpenAI AgentKit ships datasets and trace grading (https://openai.com/index/introducing-agentkit/). We only have a manual playground (components/agents/AgentChat.tsx) and per-message thumbs; there is no test table or runner anywhere under lib/agents*. A small team will not trust an agent after editing instructions without a one-click 'did I break anything' check. First scope: an agent_tests table (user message or short script, expected behavior in plain language, expected tool or handoff), a 'Tests' tab that runs all cases through runAgent in dry-run mode, judges with a cheap model and shows pass/fail with the transcript, and 'save as test' from any conversation. Priority: high. Effort: L. Compared with Intercom Fin, Zapier Agents, Lindy (plus OpenAI AgentKit as reference for evals/versioning).
Comentários 0
Ainda sem comentários.
Entre para comentar