Forward-Chaining vs Backchaining

Two modes of strategic reasoning about what to work on:

Backchaining: Start from a desired end state and reason backward through the causal steps needed. High-leverage when the mechanism from here to there is already understood.

Forward-chaining: Start from what seems tractable, generative, or interesting — what gives you real feedback and hooks into understanding — and follow where it leads.

The backchaining bias

In ambitious settings (AI safety, startups, long-term projects), there's a systematic bias toward backchaining. This produces a specific error: dismissing forward-chaining avenues because they lack a pre-specified theory of impact. But this reasoning fails for novel scientific or technical work:

"Big breakthroughs open up possibilities that are very hard to imagine before those breakthroughs — imagine trying to describe the useful applications of electricity before anyone knew what it was or how it worked; or imagine Galileo trying to justify the practical use of studying astronomy." — Richard Ngo

The whack-a-mole fallacy

Showing that each proposed mechanism from A to B fails does not rule out forward approaches. The mechanism that actually works is typically not imagined in advance. Winning whack-a-mole on current theories of impact doesn't conclusively discredit the avenue — it only rules out the theories you've thought of so far.

A forward-chaining avenue is justified when:

It provides real feedback and hooks into understanding
It represents the best available bet for principled knowledge in a domain
Breakthroughs in the area would open currently-unimaginable possibilities

Ngo's note

Ngo also observes the direction error runs both ways: most ML researchers are too focused on forward-chaining and not enough on backchaining. The failure modes are symmetric; the value is in knowing which direction you're currently biased.

Sources

against-almost-every-theory-of-impact-of-interpretability