AI-103: Developing AI Apps and Agents on AzureDemo

By Microsoft · Question Mode

QuestionQ8

Plan and manage an Azure AI solution

You have a Microsoft Foundry project containing a customer support agent that is grounded in internal documentation.

Following a recent update, users report the following issues:

  • Some answers are not supported by the retrieved documents.
  • A small number of responses are flagged for policy violations.

You need to evaluate each issue. Which observability signal should you use for each one? An observability signal can be used once, more than once, or not at all.

Drag & Drop
Groundedness evaluation metrics
Latency breakdown traces
Risk and safety metrics
Token usage analytics
Unsupported responses:
Policy violations:
Explanation

Groundedness evaluation measures whether a generated response is actually supported by the content retrieved from the grounding documents, so it is the correct signal for catching answers that are not backed by retrieved documents. Risk and safety metrics assess generated content for policy-violating material such as harmful, unsafe, or non-compliant output, so they are the correct signal for responses flagged for policy violations. Latency breakdown traces and token usage analytics measure performance and cost, not answer quality or safety, so they do not address either issue.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!