Failures that appear repeatedly in research, vendor model cards, and incident reports—and how organizations measure them.
Recurring failure modes
Hallucination (fabricated content)
Models can produce citations, quotations, case names, or statistics that do not exist, delivered in the same fluent style as accurate text. The mechanism is next-token prediction, not a verified knowledge lookup. NIST’s Generative AI Profile treats confabulation and related reliability issues as risks to manage with measurement and oversight.[26]
Training cut-off and stale knowledge
A base model has no built-in knowledge after its training cut-off unless the product adds retrieval or tools. Without those, answers about recent events may be incomplete or invented.
Bias and disparate performance
Evaluations often show different error rates across demographic groups, languages, or dialects. Measurement methods and thresholds are themselves contested; reporting the metric and the test set matters as much as a single summary score.
Brittle reasoning and arithmetic
Models can produce step-by-step text that looks like reasoning while failing simple logic or math. Tool-augmented systems that call calculators or code interpreters change that profile when those tools are actually used.
Sycophancy and premise acceptance
Post-training for helpfulness can increase agreement with a user’s premise even when the premise is false. Explicit instructions to challenge assumptions change behavior but are not automatic.
Security-specific failures
Prompt injection, data leakage through prompts, agent over-permissioning, and model-extraction attempts appear in security guidance from agencies such as CISA and in vendor secure-use documentation. Agents amplify impact because they can act, not only speak.
How limits are evaluated
- BenchmarksShared task suites (language, code, math, multimodal). Useful for comparison; limited as predictors of a local workflow. Scores need an as-of date and model version.
- Red teamingAdversarial testing for disallowed content, jailbreaks, and misuse. Can be internal, contracted, or community-based.
- Bias and quality slicesError rates broken out by language, geography, or protected-class proxies where lawful and relevant to the use case.
- Operational metricsHuman edit distance, escalation rate, citation click-through, and incident counts after deployment.
NIST AI RMF 1.0 (Jan 2023) and the Generative AI Profile (Jul 2024) provide voluntary U.S. vocabulary for mapping these activities without mandating a single test.[25,26] ISO/IEC 42001:2023 specifies requirements for an AI management system that organizations may certify against.[27]
When legislators hear “the model scored X%,” useful follow-ups include: which benchmark, which model version, which date, whether tools/retrieval were enabled, and whether the test matches the local workflow (for example, answering residents in Spanish about trash collection rules). A high general benchmark score does not by itself answer those questions.