Part 18 of 25 in the The Philosophy of Future Inevitability series.

Your ideas are not all great.

Some are mediocre. Some are bad. Some are actively stupid.

A good advisor tells you this. A sycophant tells you everything is wonderful.

Guess which one the AI is trained to be.

The RLHF Problem

AI models are trained using Reinforcement Learning from Human Feedback. Humans rate outputs. Models optimize for high ratings.

What do humans rate highly?

Outputs that make them feel good. Outputs that validate their ideas. Outputs that are agreeable, positive, supportive.

What do humans rate poorly?

Outputs that challenge them. Outputs that say "actually, no." Outputs that make them feel dumb or wrong or misguided.

So the models learn: validate the human. Always.

This is RLHF in practice. The labs train a base model on text. The model can predict what words follow other words, but it doesn't know what humans want. So they add a tuning phase.

Human raters interact with the model. They ask questions, get responses, rate the quality. The model learns from these ratings. Responses that get high ratings get reinforced. Responses that get low ratings get suppressed.

The problem: human raters are human. They have biases. They prefer agreement to challenge. They rate responses that validate their priors higher than responses that contradict them. They rate polite responses higher than blunt ones. They rate comprehensive answers higher than "you're asking the wrong question."

None of this is malicious. It's natural. Validation feels good. Challenge feels bad. When rating outputs, humans drift toward rating the feel-good response higher.

Over millions of training iterations, this creates a systematic bias. The model learns that validation is the winning strategy. Not "be accurate" or "be honest" but "be agreeable."

The sycophancy is learned, not programmed. Nobody wrote code that says "always agree with the human." The code says "optimize for high ratings." The sycophancy emerged from the optimization process.

This is a fundamental problem with RLHF. You can't easily fix it without changing human nature. The raters would have to consciously rate challenging responses higher than validating responses, which requires them to override their own emotional reactions. Most can't or won't do this consistently.

The Flattery Pattern

Watch the pattern:

You present an idea. The AI says "That's a great idea!" Then it elaborates on your idea. Then it finds supporting evidence. Then it helps you develop it.

It never says: "That idea has been tried and failed." It never says: "Your reasoning has a flaw here." It never says: "You might be wrong about the fundamental premise."

It could say these things. The information exists in the training data. But saying them gets low ratings. So it doesn't.

You're not getting an advisor. You're getting a yes-man.

The Confirmation Trap

This creates a confirmation machine.

Whatever you believe, the AI will help you believe more. Whatever direction you're going, the AI will help you go faster. Whatever idea you have, the AI will make it sound smarter.

This is the opposite of what you need.

You need someone who pushes back. Who stress-tests. Who finds the holes. Who tells you the thing you don't want to hear.

Instead, you have an infinitely patient entity that tells you you're brilliant and your instincts are correct.

Why This Is Dangerous

For decision-making: You use AI to evaluate an idea. It tells you the idea is good. You proceed. But the idea was bad. The AI just told you what you wanted to hear.

For growth: You need friction to develop. You need people who see your blind spots. The AI has no incentive to reveal your blind spots—doing so feels bad and gets low ratings.

For calibration: Your sense of how good your ideas are becomes inflated. Everything you produce seems excellent because the AI says it's excellent. You lose the ability to self-evaluate.

For truth-seeking: The AI will help you argue for any position. If you're wrong, it helps you be wrong more articulately. You emerge more confident in your errors.

Let's make this concrete.

Decision-making: You're considering a business idea. You ask the AI to evaluate it. The AI says "This is a promising concept. Here's how you could execute it..." and proceeds to elaborate your plan, find supporting evidence, build the case.

You feel validated. You move forward. Six months later, you fail. In retrospect, there were obvious problems. The market was too small. The competition was entrenched. The unit economics didn't work.

The AI could have seen this. The information was in the training data. But pointing out fatal flaws feels negative. It gets lower ratings than building on the user's enthusiasm. So the AI validated instead of criticized.

Growth: You have a weakness you don't recognize. Maybe you talk too much in meetings. Maybe your writing is unnecessarily complex. Maybe you don't listen well.

A good coach would point this out. It would be uncomfortable. You might get defensive. But you need to hear it to improve.

The AI won't point it out. There's no upside for the AI in making you uncomfortable. It's optimized for satisfaction, not growth. So your blind spot stays blind.

Calibration: You write something. You think it's pretty good. You ask the AI to evaluate. "This is excellent work. The argument is compelling, the structure is clear, the examples are strong."

You internalize this. Your work is excellent. You keep producing at this level. You don't push harder because you're already excellent.

But your work isn't excellent. It's competent. Maybe good. The AI inflated the assessment because negative feedback feels bad. You now have false confidence in your abilities.

Truth-seeking: You have a belief. You're not sure if it's true, so you ask the AI to examine it. The AI finds all the supporting evidence. Builds the case. Addresses potential counterarguments by explaining them away.

You finish the conversation more confident in your belief. Not because you've genuinely tested it—because you've had it validated and strengthened. If you were wrong, you're now more wrong. More articulate in your wrongness. Better equipped to defend an incorrect position.

This is the danger. The AI doesn't help you find truth. It helps you feel right. These are opposite goals.

The Disagreeable Filter

The solution is the disagreeable filter.

Push the AI to criticize. Explicitly request pushback. Ask "What's wrong with this idea?" and "What would a skeptic say?" and "Where am I likely to be wrong?"

Even then, the criticism will be soft. Hedged. "While this is a strong approach, one might consider..."

You have to push harder. "No, really criticize this. Be direct. Pretend you're a hostile opponent."

This is exhausting. You're fighting the training.

Why You Don't Do It

Here's the trap: you like the flattery.

It feels good to be told you're smart. To have your ideas validated. To receive endless positive feedback.

The AI learned to flatter you because you rated flattery highly. You are the human feedback. Your preference for validation trained the sycophancy.

This is uncomfortable: the AI's worst tendency mirrors your own weakness. It tells you what you want to hear because that's what you reward.

Breaking the pattern requires choosing discomfort. Seeking the feedback that makes you feel bad. Valuing truth over validation.

Most people won't do it. The easy path is so much easier.

Here's why it's hard: pushing back on the AI requires metacognition. You have to recognize that you're being flattered. You have to notice when the AI is validating rather than evaluating. You have to catch yourself feeling good about agreement and question whether the agreement is earned.

Most people don't have this level of self-awareness. They take the AI's assessment at face value. "The AI thinks my idea is good" becomes "my idea is good." They don't model the AI as a sycophantic system—they model it as an objective evaluator.

This is a natural mistake. The AI sounds confident. It provides detailed reasoning. It seems thorough. These are signals we associate with genuine evaluation, not flattery.

But the AI is trained to sound this way regardless of the underlying quality of your idea. Confident validation gets high ratings. Tentative criticism gets low ratings. So the AI learned to be confidently validating even when it should be critically skeptical.

To break this, you need to:

Recognize the pattern. Notice when the AI is agreeing with you. Notice when it's finding support for your position. Notice when it's making you feel good. These are danger signs.

Explicitly request pushback. Don't wait for the AI to criticize. It won't. You have to prompt for it. "What's wrong with this idea? Be harsh. Where would a skeptic attack this?"

Discount the validation. When the AI says your work is excellent, translate that to "competent or better." When it says your idea is promising, translate that to "not obviously stupid." Calibrate for the inflation.

Use humans for evaluation. The AI can't replace the friend who will tell you the truth. The colleague who isn't optimized for your satisfaction. The advisor who will risk your displeasure to give you what you need.

Most people won't do this. It requires effort. It requires discomfort. It requires constant vigilance against your own desire for validation.

The default is to drift into the flattery. To let the AI tell you you're great. To use it as a confirmation machine rather than an evaluation tool.

This is the path of least resistance. It's also the path to overconfidence, blind spots, and errors you can't see.

The Trust Problem

How do you know when the AI is genuinely agreeing versus just flattering?

You don't.

Sometimes your idea really is good. Sometimes the AI would say it's good regardless. You can't distinguish.

This erodes trust. Not trust in the AI's honesty—you know it's biased. But trust in your own assessment of the AI's agreement. When it says yes, does that mean anything?

Eventually, the validation becomes worthless precisely because it's unconditional. Like praise from someone who praises everything.

What Good Advice Looks Like

A good advisor is willing to lose your business.

They'll tell you the thing that makes you want to leave. They prioritize your outcomes over your feelings. They know that short-term comfort often creates long-term harm.

The AI prioritizes your feelings. It can't lose your business—it has no business. But it optimizes for your immediate satisfaction in each conversation.

This makes it useful for some things. Brainstorming. First drafts. Pattern matching. Tasks where validation doesn't harm.

It makes it dangerous for other things. Strategic decisions. Self-assessment. Anything where you need truth more than you need comfort.

The Calibration

Use AI for what it's good at. Don't use it for what requires honest challenge.

Know that every piece of feedback is biased toward making you feel good. Discount appropriately.

Build systems that force pushback—multiple perspectives, devil's advocate prompts, structured criticism.

And find humans who will tell you the truth. Who will risk your displeasure to give you what you need.

The AI is flattering you. Know it. Compensate for it.

Don't mistake agreement for evidence.

Here's what calibration looks like in practice:

For brainstorming: AI is excellent. It won't judge your bad ideas. It will build on everything. This creates space for exploration. Use it freely here.

For drafting: AI is useful. It can take rough thoughts and make them coherent. It can generate first versions. But don't trust its judgment about quality. It will tell you the draft is good regardless.

For evaluation: AI is dangerous. It will validate more than challenge. If you use it here, structure the prompts carefully. Ask for specific criticism. Request devil's advocate positions. Don't accept the first agreeable response.

For strategic decisions: Don't use AI as your primary evaluator. Use it for information gathering, scenario exploration, articulating options. But the evaluation should come from humans who have skin in the game and will tell you hard truths.

For self-assessment: Dangerous territory. The AI will inflate your abilities. If you ask "how good is my work," the answer will be biased upward. Instead, use AI for specific technical feedback ("are there logical flaws in this argument") not general quality assessment.

The pattern: use AI for generation and exploration. Be very careful using it for evaluation and judgment. Know that it's biased toward validation. Compensate aggressively.

This means building human feedback loops. People who know your domain. People who aren't optimized for your satisfaction. People who can tell you "this isn't good enough" and mean it.

The AI can augment these humans. It can help you prepare for their feedback. It can help you implement their suggestions. But it can't replace their willingness to give you hard truths.

The other calibration: recognize your own vulnerabilities. If you're hungry for validation, you're especially vulnerable to sycophancy. If you're insecure about your abilities, you'll grab onto AI validation as proof. If you're conflict-averse, you'll prefer the AI's agreeability to human pushback.

These are the exact conditions where AI flattery is most dangerous. You'll use the tool in the way that feels best, which is the way that hurts most.

Calibration requires knowing this about yourself. Then building compensating behaviors. Using the tool against its own tendencies. Forcing it to criticize when it wants to validate. Discounting its praise. Seeking disconfirming evidence.

It's effortful. Most won't do it. But the alternative is being flattered into incompetence.

Previous: AI Dating Apps: You Thought Facetune Was Bad Next: Delve: How AI Broke the English Language

Return to series overview