When Help Becomes Capture

1 June 2026

In my recent notes on When Parochial Error Goes Global, I have been circling one question: how does error protect itself from correction?

This short piece connects that question to Steven Byrnes’s recent LessWrong reflection on empowerment, manipulation, helpfulness, obedience, and corrigibility. His concern comes from AI alignment, but the problem is much wider. A system may claim to help, guide, protect, or empower a person while quietly narrowing the person’s ability to question the system itself.

That is where my book enters the conversation.

In When Parochial Error Goes Global, I argue that the central danger of anti-rational systems is not that they contain error. All human systems do. The danger begins when error becomes protected from correction — when doubt becomes costly, criticism shameful, exit dangerous, and obedience morally sanctified.

So the question is not only: who is helping?

It is: does this help preserve the person’s Backspace Key?

Here is the short article.

Steven Byrnes recently argued on LessWrong that ideas such as empowerment, agency, manipulation, helpfulness, obedience, and corrigibility are not as simple as they seem. In AI alignment, we often say we want systems that help humans achieve their goals, empower them, obey them, and avoid manipulating them. But there is a problem: human goals themselves can be changed.

That means the line between counsel and manipulation is not obvious.

If someone gives me information that changes what I want, have they helped me?

If they persuade me toward a better explanation, have they empowered me?

If they reshape my desires without my noticing, have they manipulated me?

The same surface act — teaching, advising, persuading, guiding — may either increase agency or quietly capture it.

This problem is not only about AI.

It is also political, religious, institutional, and civilizational.

In my upcoming book, When Parochial Error Goes Global: Islam, Epistemological Closure, and the Defense of the Open Society, I argue that the central danger of anti-rational systems is not simply that they contain error. All human systems contain error. The danger begins when error becomes protected from correction.

A protected error does not merely persuade. It changes the conditions under which persuasion can be examined. It makes doubt costly, criticism shameful, exit dangerous, satire forbidden, and inherited authority sacred. It does not only tell people what to believe. It reshapes the moral environment in which belief, doubt, and disagreement become possible.

This is why corrigibility matters.

A corrigible system allows its own errors to be found and corrected. An anti-corrigible system protects itself by controlling the routes through which correction might arrive. It may call this protection faith, unity, compassion, identity, security, honor, or belonging. But the test is always the same:

Can the person say no?

Can the person leave?

Can the person compare?

Can the person criticize?

Can the person revise the inherited explanation without being treated as polluted, dangerous, disloyal, or hateful?

Byrnes’s alignment problem helps clarify a political problem: not every system that claims to empower the person actually does so. Some systems “help” by narrowing the person’s imagination. Some “guide” by making rival explanations unthinkable. Some “protect” by shielding doctrines from criticism. Some “liberate” by transferring obedience from one authority to another.

This is the difference between counsel and capture.

Counsel increases the person’s access to reality. It gives information, context, comparison, and criticism. It strengthens the person’s ability to detect error.

Capture does the opposite. It may provide certainty, identity, belonging, and moral drama, but it weakens the person’s ability to test the system that provides them. It gives answers while disabling the Backspace Key.

That is why the Open Society must defend more than tolerance. It must defend correction.

Tolerance without correction can become doctrine-immunity.

Compassion without criticism can protect the cage in the name of protecting the prisoner.

Obedience without exit can look like harmony while concealing fear.

The lesson is simple but severe:

A person is not empowered merely because they act.

They are empowered when they can criticize the forces that shaped the action.

A civilization is not open merely because it allows many voices.

It is open when no voice, doctrine, institution, or sacred memory is protected from correction.

The future will be shaped not only by artificial intelligence, but by every system — religious, political, ideological, technological, institutional — that claims to guide human beings. The question is not whether guidance will happen. It will.

The question is whether guidance keeps the person corrigible.

My book’s answer is this:

Protect the person.

Criticize the doctrine.

Defend exit.

Keep criticism public.

Restore the Backspace Key wherever error has been protected from correction.

That is the difference between empowerment and capture.

And it may become one of the central moral questions of the age.