QuestionQ75

VerifiedOperational Efficiency and Optimization for GenAI Applications

A company uses Amazon Bedrock and Anthropic Claude 3 Haiku to build an AI assistant. The AI assistant typically handles 10,000 requests per hour, but during peak-use periods it experiences surges of up to 30,000 requests per hour. The AI assistant must respond within 2 seconds while operating across multiple AWS Regions.

The company finds that, during peak-use periods, the AI assistant has throughput bottlenecks that result in increased latency and occasional request timeouts. The company must address these performance issues.

Which solution will meet this requirement?

Explanation

Amazon Bedrock cross-Region inference profiles automatically route model-invocation requests to available destination Regions, distributing traffic to increase throughput and manage unplanned traffic bursts. Token batching reduces API overhead, while keeping the workload on the synchronous inference path needed for a 2-second response target. Increase throughput with cross-Region inference

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!