AI-103: Developing AI Apps and Agents on Azure

By Microsoft · Question Mode

QuestionQ21

VerifiedImplement generative AI and agentic solutions

You have a customer support agent built with the Microsoft Foundry Agent Service. The agent calls an Azure OpenAI model deployment.

During load testing, calls intermittently fail with an HTTP 429 "rate limit exceeded" error.

You need to handle the throttling to reduce call failures and improve reliability under load. The solution must stay within the service and model limits.

What should you do?

Explanation

The recommended pattern for handling Azure OpenAI HTTP 429 responses is a client-side retry policy that uses exponential backoff combined with random jitter (honoring any Retry-After header when present). This spreads retries out over time so the client automatically slows down and stays within the deployment's rate limit instead of repeatedly hammering it, improving reliability without requesting additional quota. Retrying immediately on a new thread only compounds the throttling, and reducing tool count or splitting uploaded files does not address a token/request-rate limit.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!