Can we ever fully trust what an AI says?

The single most stubborn open problem in everyday AI use is reliability: a model can produce a fluent, confident answer that is simply wrong. This is called hallucination, and the worrying part is that a false answer sounds exactly like a true one. There is no built-in tone of doubt to warn you.

What makes this an open question rather than a solved bug is that the behavior persists even in the best systems. The 2025 paper “Why Language Models Hallucinate” notes that these errors “persist even in state-of-the-art systems and undermine trust,” and argues they “need not be mysterious” because they follow from how models are trained and graded.^[1] If guessing scores better than admitting “I don’t know,” a model learns to guess. That insight points to better grading, but it does not yet give us a model that never bluffs.

So the honest status is: hallucination can be reduced, but not removed. Giving a model the right facts to work from (see retrieval) and changing how answers are scored both help. Neither produces a guarantee. For low-stakes drafting, the current level is fine. For medicine, law, or anything where a wrong fact causes real harm, the unanswered question is whether a system built this way can ever be trusted without a human checking the output.