AWS Certified AI Practitioner AIF-C01Demo

Question

A social media company wants to use a large language model (LLM) to summarize messages. The company has chosen a few LLMs that are available on Amazon SageMaker JumpStart. The company wants to compare the generated output toxicity of these models.

Which strategy gives the company the ability to evaluate the LLMs with the LEAST operational overhead?

A Crowd-sourced evaluation
B Automatic model evaluation
C Model evaluation with human workers
D Reinforcement learning from human feedback (RLHF)

Explanation

Automatic model evaluation uses predefined metrics and tools to systematically assess model outputs for toxicity and other quality measures with minimal human intervention. This approach dramatically reduces operational overhead compared to crowd-sourcing, which requires many reviewers, or manual human review. Reinforcement learning from human feedback requires extensive human labeling, making it far more resource-intensive.

Learn more

AWS Certified AI Practitioner AIF-C01Demo

AstroTutor

Notes

Readiness

QuestionQ219

Community Discussion