The Boiling Frog

Last week more than 1,100 people at OpenAI, Anthropic, Google, and Meta signed Pacing the Frontier, a letter asking the US government to support building tools to deliberately pace frontier AI development. Both OpenAI and Anthropic endorsed it at the company level.

But what if things are actually not moving fast enough?

What if we face a boiling frog problem with the leverage that members of society have? Suppose those on the side of humanity slowly have their leverage and power taken away, bit by bit, with no single event intense enough to trigger a strong enough reaction to protect people. The alternative is a transition abrupt enough to trigger that reaction, and of course that means near-term hardship.

I talked this through with ChatGPT, and it framed the argument this way:

If AGI/ASI is inevitable, then a rapid transition may be preferable to a slow one. During a slow transition, people gradually lose economic and political leverage as machines become capable of performing more work. By the time society realizes how much power has shifted, meaningful decision-making may already be concentrated among the small number of organizations that own or control the most capable AI systems. In contrast, if transformative AI arrives while most people still retain jobs, wealth, political influence, and institutional power, they may be in a stronger position to shape the rules governing the new world. The disruption would be more severe in the short term, but it would occur while democratic institutions and broad public influence are still relatively intact.

Succinctly: power should shift while people still possess meaningful bargaining power.

I don't know what the answer is. But I don't see people pointing out this failure path, so now it's out there. It's something I'll be keeping an eye out for.