QuestionQ101

VerifiedTesting, Validation, and Troubleshooting

A company wants to select a new foundation model (FM) for its AI assistant. A GenAI developer needs to generate evaluation reports to help a data scientist assess the quality and safety of various foundation models (FMs). The data scientist provides the GenAI developer with sample prompts for evaluation. The GenAI developer wants to use Amazon Bedrock to automate report generation and evaluation.

Which solution will meet this requirement?

Explanation

Amazon Bedrock model evaluation jobs that use an LLM as a judge require the prompt dataset to be a JSONL file (one JSON object per line) stored in Amazon S3, and the job requires selecting both a generator model (the FM being tested, which responds to the prompts) and a judge/evaluator model (which scores those responses for quality and safety). The job output, including the evaluation report, is written to a specified S3 location. Running a separate evaluation job for each candidate FM, with that FM designated as the generator, produces automated comparative reports for the data scientist. QuickSight is not a supported output target for Bedrock evaluation jobs, and assigning the FM under test as the 'evaluator' (rather than generator) is incorrect, since the judge model performs the scoring, not the model being tested.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!