QuestionQ60

VerifiedOperational Efficiency and Optimization for GenAI Applications

An ecommerce company uses Amazon Bedrock to build a customer-service AI assistant. The AI assistant must process more than 50,000 customer inquiries each day. During promotional events, the AI assistant occasionally has traffic spikes of up to 150,000 inquiries per day. Analysis indicates that 40% of inquiries follow similar patterns and share the same context.

A GenAI developer must design a solution that ensures low latency and consistent performance for the AI assistant during traffic spikes.

Which solution will meet these requirements in the MOST cost-effective way?

Explanation

Amazon Bedrock latency-optimized inference provides faster responses by setting the request performance configuration latency value to optimized. Prompt caching reuses static, repeated prompt context, avoiding recomputation and reducing both inference latency and input-token costs. This is well suited to the 40% of inquiries that share context, while avoiding the ongoing expense of provisioned throughput sized for infrequent peak demand.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!