Model-graded evaluations use a language model to judge agent responses against qualitative criteria and can assess several dimensions of response quality. Deterministic evaluations instead apply fixed, reproducible rules such as string comparisons or programmatic checks. OpenAI’s grader reference defines score-model and label-model graders as graders that use a model to assign scores or labels, while string-check graders perform specified string comparisons.
Community Discussion