QuestionQ173

Domain IV — Understanding how to govern AI deployment and use

Business A sells software that gives users writing and grammar assistance. Business B is a cloud-services provider that trains its own AI models.

  • Business A has decided to incorporate generative AI features into its software.
  • Instead of developing its own generative AI model, Business A has elected to license a model from Business B.
  • Business A will integrate that model into its writing-assistance software to deliver generative AI capabilities.
  • Business A is primarily concerned that its writing-assistance software might recommend toxic or obscene text to users.

Which governance process should Business A use to best protect its users from potentially inappropriate text?

  • A Business A should fine-tune the AI model on user-generated text that has been verified to be appropriate.
  • B Business A should test that the AI model performs as expected and meets their minimum requirements for filtering toxic or obscene text.
  • C Business A should establish a user reporting feature that allows users to flag toxic or obscene text, and report any incidents to Business B.
  • D Business A should ask Business B for detailed documentation on the generative AI model’s training data and whether it contained toxic or obscene sources.
Explanation

Pre-deployment testing and evaluation against explicit content-safety requirements verifies whether the deployed AI system filters toxic or obscene outputs to the organization’s required threshold. This is a direct preventive control for a known output-risk; incident reporting and training-data documentation are useful complementary measures but do not establish that the integrated system meets the required filtering performance. NIST’s AI Risk Management Framework calls for objective testing, evaluation, verification, and validation processes and for the system to be evaluated for identified safety risks.

Learn more

Community Discussion

No comments yet. Be the first to start the discussion!