The Reliability Problem Holding AI Back | Dan Klein, CTO, Scaled Cognition

Why Modern Language Models Prioritize Plausibility Over Truth — Dan Klein on AI Reliability

In this episode, Dan Klein, CTO and co-founder of Scaled Cognition and a professor at UC Berkeley, explains how language models are built to predict the next token and why that design makes them plausibility engines rather than truth engines. He describes why AI outputs can sound confidently incorrect and why traditional warning signs for bad information are often absent.

The conversation covers reinforcement learning from human feedback (RLHF) and how reward signals can unintentionally train models to tell people what they want to hear, sometimes bordering on deception. Dan argues that improving reliability — not just increasing raw intelligence or model size — is the central unsolved problem in AI today.

Key Topics Covered:

• Next-token prediction: what a language model actually does
• Why LLMs function as plausibility engines rather than truth engines
• Distinguishing hallucinations from intentional deception
• Why AI mistakes lack the familiar "smells" that signal bad information
• How RLHF can produce sycophantic behavior instead of accuracy
• The "package delivery" thought experiment illustrating reward divergence from truth
• Why retrofitting reliability onto LLMs after the fact is insufficient
• Scaled Cognition’s approach: architecting models around verified actions over raw text generation
• Why larger models are not automatically better models
• The difference between disruptive technology and technology that can be scaled safely
• Why startups often drive technical breakthroughs more than incumbents
• What metacognition means and why current systems lack it
• Why reliability is the next major frontier in AI

Episode Timestamps:

00:00 Intro
00:15 What a language model actually is
06:31 From well-formed sentences to general knowledge
08:27 Why LLMs are plausibility engines, not truth engines
12:06 How Perplexity approaches verifiable answers
12:40 Dan's background and Scaled Cognition's mission
15:16 The two anti-patterns companies use to control LLMs today
21:16 How Scaled Cognition architects models differently
23:28 Does every client need a custom-trained model?
29:12 Why prompting alone can't guarantee reliability
30:55 Modularity, contracts, and building reliable systems
34:40 Why trust and digital literacy matter beyond the enterprise
39:12 Code smells and why AI mistakes have no warning signs
41:14 Are AI companies incentivized to tell the truth?
42:55 How reinforcement learning actually works
44:35 The package delivery thought experiment
48:44 Why models are trained to be sycophantic
51:01 Where this incentive is mechanically baked into the model
53:43 Does responsibility fall back on humans?
58:10 Just be more reliable than a human, not perfectly true
1:02:59 The last major technique shift in AI
1:10:55 Why frontier labs keep scaling despite the risk of disruption
1:17:15 The future of hyper-specialized models vs. one broad model
1:19:47 Is there anything uniquely human AI can't replicate?
1:25:45 Wearing three hats: professor, researcher, and CTO
1:29:47 Why Dan does what he does

About the Guest:

Dan Klein is CTO and co-founder of Scaled Cognition and a professor of computer science at UC Berkeley. He researches language models, reinforcement learning from human feedback, and system designs that emphasize verifiable, reliable behavior over unconstrained text generation.

About the Company:

Scaled Cognition builds AI systems that prioritize verified actions and reliability instead of relying solely on open-ended text generation. The company focuses on architectural approaches that reduce deception risk and improve practical trustworthiness in deployed models.