Model Cards

Description

Publish a short structured document with each model stating how it was built and evaluated, its intended uses and users, performance across groups, and its limitations.

A model card accompanies a trained model and discloses what stakeholders need to judge whether it fits their purpose, covering ethical and fairness considerations as well as metrics. Standard sections are model details; intended use, including primary uses, users and out-of-scope uses; factors (demographic, environmental, technical); metrics, decision thresholds and variation; evaluation data; training data; quantitative analyses, both unitary and intersectional; ethical considerations; and caveats and recommendations. Cards sit within a wider responsible-AI process: finding sources of bias at each stage, understanding data limits with domain experts and affected people, and weighing trade-offs between privacy, compression and fairness. Because cards must be updated with every model version, generation should be automated. For UX, the intended-use, factors and caveats sections are where user research about contexts and affected groups becomes a durable, checkable commitment.

Attributes

Loss type
Siloed
Data
Mixed
Effort
Days

Steps

  1. Review sources of bias across training data, labelling, features, objective and evaluation.
  2. Run fine-grained evaluation sliced by relevant groups and their intersections.
  3. Fill each section: model details, intended use, factors, metrics, evaluation data, training data, quantitative analyses, ethical considerations, caveats and recommendations.
  4. Involve domain experts and affected users in drafting intended and out-of-scope uses.
  5. Publish the card with the model and link it from product documentation where users or customers can see it.
  6. Regenerate the card on every model update, ideally automatically from the evaluation pipeline.

Sources

  • Designing Machine Learning Systems (Huyen, 2022)

Also known as

  • Model cards for model reporting
  • Model documentation

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning