QuestionQ115
VerifiedOperational Efficiency and Optimization for GenAI ApplicationsA publishing company is building a chat assistant powered by a containerized large language model (LLM) that runs on Amazon SageMaker AI. The architecture includes an Amazon API Gateway REST API that forwards user requests to an AWS Lambda function, which in turn invokes a SageMaker AI real-time endpoint hosting the LLM.
Users are experiencing inconsistent response times. Analytics reveal that many chats are abandoned after users wait 2 seconds for the first token to appear. The company needs a solution that keeps p95 latency under 800 ms for interactive requests to the chat assistant.
Which combination of solutions will meet this requirement?
Choose two
Community Discussion