Start here

Home › Myths vs Facts › Is AI always right if it sounds sure?

Myths vs facts

Is AI always right if it sounds sure?

No. Chatbots can state false things in a confident voice. NIST calls this confabulation, and in 2023 a federal court penalized lawyers who filed court cases that ChatGPT had made up.

Checked on Oct 2, 2026

Short answer

ConfirmedMyth

$5,000

penalty in a 2023 federal court order over made-up AI case citations

The evidence

What the experts call it. NIST defines confabulation as “the production of confidently stated but erroneous or false content,” known colloquially as “hallucinations” or “fabrications.”1 NIST says confabulations are a natural result of how generative models are designed: they generate outputs that approximate the statistical distribution of their training data, and language models predict the next word. Such prediction can produce accurate output, but it can also produce inaccurate output.1

Why a sure tone misleads. NIST says risks arise when users believe false content, often because of the confident nature of the response, and that the output can also include confabulated logic or citations that make the answer look justified.1

The makers agree. OpenAI defines hallucinations as plausible but false statements generated by language models, and says they remain a fundamental challenge for all large language models. It says newer models hallucinate significantly less, especially when reasoning, but that the problem still occurs. That is a company claim.3

A real case. In Mata v. Avianca, the court found that two lawyers and their firm submitted non-existent judicial opinions with fake quotes and citations created by ChatGPT, then stood by them after the court questioned them. On June 22, 2023 the court imposed a $5,000 penalty, jointly on the lawyers and the firm.2

What to do. Ask for the source, then open it yourself. See How AI works and Prompting: check the answer.

What we don’t know

  • How often any particular chatbot is wrong is not answered here. This page cites no error-rate figure, and OpenAI’s own statement that rates have fallen is a company claim.
  • Models keep changing, so a limitation described in 2023 or 2024 may look different in the newest versions. The sources still say it has not gone away.
  • This page does not cover how often people are fooled in everyday use. The court case is one documented example, not a measure of how common the problem is.
About the label. Confirmed means we read a primary source for the figures on this page. Company claims and projections are still marked as such. Our own arithmetic and opinions are marked as ours. Found a mistake? mike@ohmyai.ai. The short versions of all eight myths are on Myths vs Facts.

Sources for this page

  1. Artificial Intelligence Risk Management Framework: Generative AI Profile (NIST AI 600-1). National Institute of Standards and Technology.Published July 2024 · text checked Oct 2, 2026
  2. Mata v. Avianca, Inc., No. 22-cv-1461 (PKC), Opinion and Order on Sanctions (Document 54). U.S. District Court, S.D.N.Y. (copy on Justia).Filed June 22, 2023 · checked Oct 2, 2026
  3. Why language models hallucinate. OpenAI.Published Sept 5, 2025 · checked Oct 2, 2026