QuestionQ53

VerifiedEvaluating and deploying agentic workflows

You are evaluating a policy question-answering agent with an Agent Development Kit (ADK) eval set. Each case stores one reference answer, and the ADK eval suite scores responses using the ADK response_match_score criterion. Reviewers have been asked to evaluate the agent’s responses and identify acceptable responses that cite the policy section and provide any exception applicable to the prompt. However, reviewers cannot agree on which answer to store because several substantially different answers are acceptable.

You want ADK evaluations to assign a pass verdict to every response that meets both conditions and a fail verdict to the rest.

What should you do?

Explanation

Define rubrics for citing the policy section and providing any applicable exception, and use rubric_based_final_response_quality_v1 to evaluate responses against them without relying on one reference answer. Set the combined-score threshold to 1.0 to require both rubrics to pass. The criterion uses an LLM judge, so its verdicts are not guaranteed to be error-free. (github.com)

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!