AI-103: Developing AI Apps and Agents on AzureDemo

A content filtering/severity threshold set to 'High' is the most permissive configuration — it blocks only the most severe (high-severity) content, while a 'Low' threshold is the strictest, blocking content at low, medium, and high severity levels. Because the fine-tuning job already fails RAI checks while the filter is at the permissive 'High' setting, the training data contains content severe enough to trigger blocking even under minimal filtering; switching to 'Low' would only make filtering stricter (catching more content), so it would not resolve — and would likely worsen — the failures. Additionally, per Microsoft's fine-tuning safety evaluation documentation, the RAI checks performed during fine-tuning use predefined thresholds that are not tied to any custom content filtering configuration, so changing a filter configuration doesn't affect this evaluation at all.

Content harm defect rate is defined as the percentage of dataset responses whose severity score exceeds the configured threshold. A medium threshold (used in Run1) is more sensitive and flags content at medium and high severity, resulting in a higher defect rate (12%); a high threshold (used in Run2) only flags the most severe content, resulting in a lower defect rate (4%). This drop is exactly the expected behavior when moving from a stricter to a more lenient threshold, confirming the difference is consistent with the threshold change.

The protected material evaluator is a binary classifier that detects the presence of copyrighted text (e.g., song lyrics, articles) in model output; it operates independently of the configurable severity scale used for harmful content categories (hate, sexual, violence, self-harm). Because the severity threshold setting has no bearing on protected material detection, it is expected that Run1 and Run2 — which differ only in harmful content severity threshold — produce the same protected material evaluation result (6% in both).