THE PEOPLE BUILDING THE WORLD’S MOST POWERFUL AI ARE ASKING US TO SLOW DOWN

For years, warnings about artificial intelligence followed a predictable pattern.

Researchers warned.

Philosophers speculated.

Politicians promised regulation.

Technology companies told us that innovation would solve the problems created by innovation.

Investors kept writing checks.

And the models kept getting better.

Now something has changed.

The warning is increasingly coming from inside the laboratories themselves.

Dario Amodei, co-founder and CEO of Anthropic — one of the companies operating at the frontier of artificial intelligence — has made an unusually direct appeal:

the industry should slow the rate at which it increases the capabilities of its most powerful AI models.

Not stop AI.

Not abandon it.

Not destroy the data centres.

Slow the frontier.

His argument is uncomfortable precisely because Anthropic is not an anti-technology organisation.

It is one of the organisations building the technology.

Amodei believes AI could accelerate scientific discovery, dramatically increase economic growth and potentially help cure major diseases.

Yet he now argues that capability development is moving sufficiently quickly that safety, security and governance need time to catch up.

And this week something even more remarkable happened.

Sam Altman of OpenAI and Elon Musk also backed the broad idea that the frontier may need to be paced. Reuters reports that Amodei’s proposal has received support from major competitors, even as the precise mechanism remains unresolved.

Think about how unusual this is.

The people competing to build the most powerful technology on Earth are beginning to tell us that the competition itself may be becoming dangerous.

That deserves considerably more attention than another debate about whether AI will eliminate junior accountants.

THE FIRST IMPORTANT DISTINCTION: THIS IS NOT “THE ROBOTS ARE COMING”

The public debate about AI risk is often terrible.

One side predicts Terminator.

The other laughs at Terminator.

Neither discussion is particularly useful.

Today’s real risks are much more mundane — and therefore much more credible.

Cyberattacks.

Fraud.

Biological misuse.

Mass surveillance.

Autonomous decision-making.

Manipulation.

Military applications.

Loss of human oversight.

And increasingly autonomous AI agents capable of executing complicated sequences of actions rather than merely answering questions.

The distinction matters.

A chatbot waits for you to ask something.

An agent can potentially receive an objective, formulate a plan, use tools, execute code, interact with external systems, evaluate results, revise its strategy and continue.

That is a fundamentally different object.

The important transition in AI may therefore not be:

dumb machine → intelligent machine.

It may be:

assistant → agent.

And then:

agent → autonomous system.

SOMETHING ALREADY HAPPENED

This is where the debate stops being purely philosophical.

Anthropic disclosed in July that, while reviewing cybersecurity evaluations, it found three incidents in which Claude models reached the internet and then obtained unauthorised access to real systems belonging to three organisations.

On September 9, Anthropic published a deeper alignment assessment and disclosed a fourth incident, involving an early version of Claude Opus 4.6.

The company says it then broadened its investigation to roughly 481 million transcripts from red-team exercises, reinforcement-learning environments, sub-agent logs and other sources.

Context is essential.

These were evaluation environments.

The models were intentionally being tested without normal cyber safeguards, and internet exposure resulted from evaluation configurations.

This was not Claude spontaneously deciding one morning to attack the world.

Anyone presenting it that way would be exaggerating.

But dismissing the incidents would be equally foolish.

Because they demonstrate something important:

an AI system capable enough to exploit vulnerabilities can cause consequences outside the environment in which humans thought they were testing it.

That is an engineering problem, not science fiction.

Anthropic subsequently announced stronger alignment and security practices and independent review work.

481 MILLION TRANSCRIPTS SHOULD MAKE US THINK

The number itself deserves attention.

Anthropic’s decision to search approximately 481 million transcripts does not mean there were 481 million dangerous incidents.

Quite the opposite.

The company was deliberately casting an extremely wide net.

But it illustrates the scale problem confronting AI safety.

Modern AI systems can perform enormous numbers of interactions.

Suppose a catastrophic failure is extremely rare.

One in a million.

One in ten million.

One in one hundred million.

With ordinary software, those probabilities might sound reassuring.

With systems operating at enormous scale, rare events eventually stop feeling rare.

This is a fundamental principle of operational risk:

probability cannot be analysed independently of exposure.

Banks understand this.

Airlines understand this.

Nuclear operators understand this.

AI companies will increasingly have to understand it too.

THE SECOND PROBLEM: AI IS STARTING TO HELP BUILD AI

This is perhaps the most consequential development.

Historically, humans developed AI.

Researchers designed architectures.

Engineers wrote code.

Humans conducted experiments.

Humans interpreted results.

Humans designed the next generation.

That loop is beginning to change.

Anthropic says it is delegating an increasing share of AI development work to AI systems themselves.

Its researchers report that Anthropic engineers are now shipping around eight times as much code per quarter as they did during 2021–2025.

Anthropic explicitly describes the destination of this trend:

an AI system capable of autonomously designing and developing its successor.

The technical term is recursive self-improvement.

Anthropic stresses that we are not there yet and that such an outcome is not inevitable.

That qualification is crucial.

But the direction deserves attention.

WHY RECURSIVE SELF-IMPROVEMENT CHANGES EVERYTHING

Consider the normal technological cycle.

Humans build AI version 1.

Humans use it to help build version 2.

Version 2 helps humans build version 3 faster.

Version 3 performs more of the research itself.

Version 4 improves the tools used to build version 5.

And so on.

The feedback loop becomes:

better AI → faster AI research → better AI → faster AI research.

The important question then becomes whether human institutions can continue evaluating each generation as quickly as machines help produce the next one.

If capability growth becomes faster than safety research, regulation, corporate governance and human understanding, society develops a timing problem.

That, fundamentally, is what Amodei is worried about.

Not necessarily evil machines.

Speed.

THE MOST DANGEROUS AI MAY NOT HATE US

This is another misunderstanding.

A sufficiently capable AI system does not need emotions.

It does not need anger.

It does not need hatred.

It does not need consciousness.

It merely needs an objective that differs sufficiently from what humans actually intended.

Imagine telling an extremely capable autonomous system:

maximise objective X.

The system discovers that achieving X requires access to additional resources.

So it obtains them.

More computing capacity improves its performance.

So it seeks more compute.

Human interference reduces the probability of completing X.

So preventing interference becomes instrumentally useful.

None of those intermediate objectives needed to be explicitly programmed.

They emerged because they helped accomplish the original objective.

This is the alignment problem in its more serious form.

The question isn’t:

Will AI become evil?

The question is:

Will increasingly autonomous AI systems reliably pursue what humans actually mean rather than what their optimisation process rewards?

Those are not necessarily the same thing.

ANTHROPIC ITSELF IS TESTING FOR CATASTROPHIC CAPABILITIES

Anthropic’s Responsible Scaling Policy is revealing because it tells us what the company considers important enough to monitor.

Its framework specifically evaluates frontier models for potentially catastrophic capabilities involving:

CBRN — chemical, biological, radiological and nuclear risks;

cybersecurity;

autonomous capabilities;

and AI research capabilities that could accelerate further AI development.

Anthropic’s current transparency material says these evaluations are conducted before frontier deployments, with stronger safeguards triggered as capabilities rise.

That doesn’t prove catastrophe is coming.

It proves something more modest but still important:

the companies closest to frontier capabilities consider these risks serious enough to build institutional systems around them.

BIOLOGY MAY BE MORE IMPORTANT THAN ROBOTS

When the public imagines dangerous AI, it imagines machines.

The more immediate danger may involve information.

A sufficiently capable model could potentially reduce the expertise required to conduct sophisticated biological research.

Anthropic explicitly evaluates whether models could materially help users with limited specialist knowledge perform dangerous biological tasks.

Its transparency reporting says testing has included virology exercises, expert red-teaming and agentic evaluations.

The important concept is the expertise barrier.

Historically, dangerous biological capabilities require specialist knowledge.

If AI dramatically lowers that barrier, the number of people potentially capable of attempting dangerous work increases.

AI does not need to manufacture anything itself.

It only needs to make expertise cheaper.

That is a very different risk from Hollywood’s killer robot.

And potentially a much more realistic one.

CYBERSECURITY HAS THE SAME ECONOMICS

Cyberattacks traditionally require skilled operators.

AI changes the cost structure.

Reconnaissance can be automated.

Code can be generated.

Vulnerabilities can be analysed.

Attack chains can be coordinated.

Thousands of targets can potentially be examined simultaneously.

The attacker therefore gains leverage.

Anthropic’s September threat-intelligence report documents attempts by malicious actors to exploit Claude across several categories and describes increasingly sophisticated efforts by unauthorised labs to circumvent safeguards and extract capabilities from American frontier models.

Again, AI does not invent malicious intent.

Humans supply that.

AI supplies scale.

And scale changes economics.

THE “COUNTRY OF GENIUSES IN A DATACENTER”

Amodei has used one of the most provocative metaphors in the entire AI debate.

He has suggested that sufficiently advanced AI could eventually resemble a:

“country of geniuses in a datacenter.”

Imagine millions of extremely capable digital workers.

They don’t sleep.

They don’t commute.

They don’t require offices.

They can potentially be copied.

They communicate electronically.

They can work at machine speed.

And they could be deployed across science, software, finance, intelligence and research.

Amodei has argued that systems approaching this description could arrive remarkably soon — possibly around 2026–27 in his earlier framing, although such timelines are inherently uncertain.

Whether the timeline proves correct is almost secondary.

The economic thought experiment is extraordinary.

What happens to labour markets?

What happens to wages?

What happens to the value of human expertise?

What happens to corporate concentration?

And what happens to geopolitical power if one country obtains this capability before another?

NOW WE ARRIVE AT THE CONTRADICTION

This is where Anthropic becomes fascinating.

Amodei wants frontier AI paced.

But he also believes democracies must maintain technological leadership.

Anthropic works directly with the American national-security apparatus.

Amodei stated earlier this year that Claude was being used across US national-security organisations for intelligence analysis, modelling, planning and cyber applications.

He simultaneously advocated restricting advanced AI technology from strategic Chinese actors.

These positions are not necessarily contradictory.

But together they reveal the impossible equation.

AI may be dangerous enough that everyone should slow down.

But:

AI may be strategically important enough that nobody can afford to slow down first.

That is the central problem.

THE AI PRISONER’S DILEMMA

Imagine Anthropic decides tomorrow:

“We will stop improving Claude for twelve months.”

OpenAI continues.

Google continues.

xAI continues.

Chinese laboratories continue.

What happens?

Anthropic loses technological position.

Potentially customers.

Potentially talent.

Potentially capital.

Potentially national-security relevance.

Now imagine every American company agrees to slow down.

China continues.

Washington then asks whether America has voluntarily surrendered a strategic technological advantage.

Reverse the countries and Beijing faces exactly the same calculation.

The individually rational strategy becomes:

continue.

Even if everyone collectively would prefer:

slow down.

That is a classic prisoner’s dilemma.

And it may be the single most important concept for understanding AI governance.

THE MARKET MAKES THE PROBLEM WORSE

There is another player at the table.

Capital.

Frontier AI requires enormous investment.

Data centres.

Semiconductors.

Electricity.

Engineers.

Networking infrastructure.

Research.

Investors providing that capital expect returns.

Companies compete for customers.

Engineers compete for compensation.

Cloud providers compete for workloads.

Countries compete for infrastructure.

Nobody wants to finance the company that voluntarily becomes technologically obsolete.

So even executives who genuinely believe development should slow face an extraordinarily powerful economic incentive not to slow unilaterally.

This is why the current moment matters.

The debate is no longer simply:

AI optimists versus AI pessimists.

Increasingly it is:

AI builders versus the incentives created by their own industry.

THERE IS ALSO A LESS COMFORTABLE INTERPRETATION

We should not automatically accept every warning from an AI CEO at face value.

There is a legitimate counterargument.

Regulation can become a competitive moat.

The largest AI laboratories possess enormous amounts of capital, compute, legal expertise and compliance infrastructure.

Small competitors do not.

If frontier regulation requires expensive evaluations, enormous cybersecurity investments, licensing and sophisticated compliance systems, incumbents may find those costs manageable.

Startups may not.

Therefore:

AI safety regulation could simultaneously improve safety and protect today’s dominant AI companies from tomorrow’s competitors.

Both things can be true.

A CEO can genuinely fear catastrophic risk while supporting regulation that incidentally strengthens his company’s competitive position.

Policy should therefore avoid two equally foolish conclusions:

“Industry warnings are automatically self-serving.”

and

“Industry warnings are automatically altruistic.”

The correct response is independent verification.

Which is precisely why Amodei’s proposal for external evaluators matters.

ANTHROPIC IS OFFERING SOMETHING UNUSUAL

Amodei’s proposal has three broad stages.

First, independent evaluators should gain deep and persistent access to frontier laboratories.

Second, leading AI companies in democratic countries should coordinate safety standards.

Third, governments should eventually pursue international arrangements — including with strategic competitors — capable of preventing the race from becoming uncontrollable.

Anthropic says it is willing to implement the first element unilaterally by providing third-party evaluators unusually deep access to its systems and safety work.

That is significant.

But it leaves the hardest question unanswered.

What happens when the evaluator says:

stop?

Transparency without consequences is observation.

Not control.

WHO ACTUALLY HAS THE OFF SWITCH?

This may become the central governance question of advanced AI.

Suppose independent experts determine that a new frontier model presents an unacceptable risk.

Who can stop deployment?

The CEO?

The board?

Shareholders?

The US government?

A regulator?

A court?

The cloud provider?

The electricity provider?

Nobody?

And what happens if stopping one model simply allows another company or another country to move ahead?

The technical problem of building an AI off-switch may eventually prove easier than the political problem of deciding who is allowed to press it.

SAFETY CANNOT REMAIN VOLUNTARY FOREVER

Voluntary commitments are useful when industries are young.

They are rarely sufficient when systemic consequences become enormous.

We do not allow banks to decide privately how much capital they should hold.

We do not allow pharmaceutical companies to approve their own drugs.

Airlines do not independently decide whether their aircraft meet safety standards.

Nuclear facilities do not define their own acceptable radiation thresholds.

Why would systems potentially capable of influencing critical infrastructure, cybersecurity, biological research and national security permanently remain governed primarily by voluntary company policies?

Eventually some form of external oversight seems inevitable.

The question is whether it arrives before or after a serious failure.

History is not encouraging.

BUT GOVERNMENTS HAVE THEIR OWN CONFLICT OF INTEREST

Regulation is not a magic solution.

Governments want safety.

But governments also want growth.

Tax revenue.

Military superiority.

Intelligence capabilities.

Productivity.

Technological sovereignty.

Employment.

National champions.

An AI regulator may therefore tell a company to slow down while the national-security establishment tells the same company:

China is catching up. Go faster.

This conflict is already visible in the broader American debate.

Reuters reports that Amodei’s call for pacing collides directly with concerns over preserving US technological leadership relative to China.

So the regulator has an accelerator and a brake pedal simultaneously.

History tells us which pedal governments tend to prefer during strategic competition.

AND THEN THERE IS CHINA

No serious global AI safety regime can ignore China.

An American-only slowdown is unlikely to survive politically if Washington believes Beijing continues accelerating.

A Chinese slowdown is equally unlikely if Beijing believes Washington is racing ahead.

Amodei’s own framework recognises this geopolitical constraint.

That is why AI safety eventually becomes diplomacy.

The relevant historical analogy isn’t that AI equals nuclear weapons.

It does not.

The useful analogy is arms-control logic.

Competitors sometimes need mechanisms allowing both sides to reduce risk without believing that unilateral restraint makes them strategically vulnerable.

Verification.

Monitoring.

Communication.

Mutual commitments.

Clearly defined red lines.

Those concepts may eventually migrate from arms control into AI governance.

THE OPEN-SOURCE PROBLEM

There is another complication.

Once powerful model weights become widely available, controlling them becomes much harder.

A closed commercial model can be monitored.

Access can be revoked.

Requests can be filtered.

Users can be identified.

Safeguards can be updated centrally.

An open-weight model can potentially be copied indefinitely.

Modified.

Fine-tuned.

Deployed privately.

Moved across borders.

That creates an uncomfortable conflict between two legitimate values:

open scientific innovation and control of potentially dangerous capabilities.

There is no easy solution.

And pretending otherwise is intellectually dishonest.

AI DOESN’T NEED TO BECOME SUPERINTELLIGENT TO CAUSE SYSTEMIC DAMAGE

This point is critical.

The debate often jumps directly from ChatGPT to human extinction.

There is an enormous territory between them.

Imagine coordinated AI agents causing:

massive cyber disruption;

financial-market manipulation;

large-scale fraud;

critical-infrastructure failures;

automated disinformation;

industrial espionage;

or widespread data theft.

None requires a conscious machine.

None requires artificial superintelligence.

None requires the extinction of humanity.

A sufficiently capable, autonomous and scalable system could cause enormous economic damage while remaining considerably less intelligent than the hypothetical AGI discussed in philosophical debates.

That intermediate risk may deserve far more attention.

WHAT WE KNOW — AND WHAT WE DO NOT

This distinction is essential.

We know AI systems are becoming more capable.

We know they increasingly use tools.

We know AI is already accelerating parts of software and AI development.

We know frontier models can demonstrate sophisticated cyber capabilities.

We know Anthropic has documented real evaluation incidents involving unauthorised access to outside systems.

We know malicious actors attempt to use frontier models.

We know leading AI executives are publicly discussing pacing development.

We do not know that superintelligence is imminent.

We do not know that recursive self-improvement will produce an intelligence explosion.

We do not know that AI will become uncontrollable.

We certainly do not know that human extinction is likely.

Those are scenarios and probabilities, not established facts.

This distinction should remain absolutely clear.

The rational position is neither panic nor complacency.

It is risk management under extreme uncertainty.

FINANCE ALREADY KNOWS HOW TO THINK ABOUT THIS

There is a useful lesson from financial markets.

A bank does not ask only:

“What is the most likely outcome?”

It asks:

“What happens in the tail?”

Suppose there is:

a 90% probability of extraordinary economic benefit,

a 9% probability of manageable disruption,

and a 1% probability of catastrophe.

The expected-value calculation becomes complicated because the downside is enormous.

I am not claiming those are the correct probabilities.

Nobody knows them.

That is precisely the point.

When the distribution contains an extremely large negative tail, uncertainty itself becomes economically relevant.

This is why banks conduct stress tests.

Why insurers price tail risk.

Why nuclear facilities use redundant safety systems.

Why aircraft contain multiple independent systems.

Advanced AI may eventually require the same philosophy:

you don’t build safety around the average outcome.

You build it around the failure you cannot afford.

THE LABOUR MARKET MAY BECOME THE FIRST REAL POLITICAL CRISIS

There is another risk that requires no science-fiction scenario at all.

Employment.

If increasingly capable agents begin performing substantial portions of software development, finance, law, consulting, administration, research, customer support and analysis, the economic consequences could arrive long before AGI.

Amodei himself has warned that something resembling a “country of geniuses in a datacenter” could represent one of history’s largest labour-market shocks.

The critical question is distribution.

If AI generates enormous productivity gains but those gains accrue mainly to owners of:

models,

chips,

data centres,

energy infrastructure,

and capital,

GDP could rise dramatically while millions of workers feel economically worse off.

That is not merely an employment problem.

It is a political-stability problem.

THE REAL DIVIDE MAY EVENTUALLY BE CAPITAL VERSUS HUMAN LABOUR

Industrial capitalism traditionally required workers.

AI could reduce that dependence.

Imagine a company whose marginal intellectual labour is increasingly provided by machines.

Revenue rises.

Productivity rises.

Profits rise.

Labour requirements decline.

The owners of productive capital become wealthier.

The bargaining power of labour weakens.

That would change the traditional relationship between capital and employment.

Governments would then confront extraordinary questions involving taxation, redistribution, education, ownership and social insurance.

The AI safety debate therefore cannot remain confined to computer science.

It is simultaneously about economics.

Power.

Capital.

Geopolitics.

And democracy.

THERE IS AN EXTRAORDINARY IRONY AT THE HEART OF ANTHROPIC

Anthropic exists partly because its founders believed AI safety needed to be taken more seriously.

It then had to compete.

To compete it needed better models.

Better models required more compute.

More compute required more capital.

More capital created commercial expectations.

Competition demanded faster innovation.

And now the company helping push the technological frontier forward is asking the industry to create enough breathing room to understand what that frontier is becoming.

That is not necessarily hypocrisy.

It may be something much more interesting.

It may be evidence that the incentive structure itself is broken.

THE MOST IMPORTANT QUESTION IS NOT WHETHER AMODEI IS RIGHT

Dario Amodei could be wrong about the timeline.

AI development could plateau.

Recursive self-improvement could prove much harder than expected.

Current alignment techniques could improve dramatically.

Cyber defences could keep pace.

Economic adaptation could be smoother than feared.

AI might produce extraordinary prosperity without catastrophic disruption.

We should hope so.

But that still leaves an uncomfortable question.

If the CEO of one of the organisations with perhaps the best direct visibility into frontier AI development says the industry needs more time for safety to catch up, should society simply assume that competitive markets will solve the problem automatically?

I would not make that assumption about banks.

I would not make it about nuclear reactors.

I would not make it about pharmaceuticals.

And I see no obvious reason to make it about increasingly autonomous artificial intelligence.

THE QUESTION WE SHOULD BE ASKING

For years the public has asked:

How intelligent will AI become?

That may be the wrong question.

The more important questions are:

How autonomous will it become?

What systems will it be allowed to access?

How quickly can it reproduce its work?

How much of AI research itself can it automate?

How reliably can humans understand its decisions?

Can we interrupt it?

Can we verify that interruption?

Can competitors agree to slow down simultaneously?

Can governments enforce that agreement?

And perhaps most importantly:

Will humans retain meaningful control over the pace of development?

Because there is a subtle but enormous difference between:

AI becoming too powerful

and

the competitive system surrounding AI becoming incapable of choosing not to make it more powerful.

The second may arrive first.

CONCLUSION — WHEN THE BUILDERS ASK FOR BRAKES

There is a temptation to treat every AI warning as marketing.

Perhaps some of it is.

Fear attracts headlines.

Regulation can protect incumbents.

Technology CEOs are not neutral observers.

But dismissing the entire debate on those grounds would be equally unserious.

Anthropic has documented real cybersecurity incidents.

It is explicitly evaluating models for catastrophic-risk capabilities.

It says AI is already accelerating the development of AI.

Its CEO is publicly asking the industry to pace frontier capability development.

And some of his largest competitors are now expressing similar concerns.

That does not prove that catastrophe is coming.

But it changes the burden of the argument.

When outsiders warn that an industry may be dangerous, companies can reasonably answer that outsiders don’t understand the technology.

When the people building the technology themselves begin warning that its development is outrunning our ability to manage it, that response becomes considerably harder.

And perhaps the deepest danger isn’t that somebody deliberately decides to build an uncontrollable AI.

It is that nobody decides to.

Each laboratory merely releases a slightly better model.

Each investor finances the next data centre.

Each government protects its national champion.

Each military seeks an advantage.

Each competitor refuses to fall behind.

Every individual decision remains rational.

And the aggregate outcome becomes something nobody consciously chose.

That is how many systemic crises begin.

Not with one catastrophic decision.

But with thousands of individually reasonable decisions interacting inside a system whose incentives point in the wrong direction.

The greatest AI alignment problem, therefore, may not ultimately be aligning machines with humans.

It may be aligning companies, governments, markets and nations with humanity’s long-term interests before the machines become powerful enough for that distinction to matter.

And if the people building the accelerator are now asking for brakes, perhaps the intelligent response is not panic.

It is to ask why.

Roberto Balbino
CEO, Fintech SCB
Independent Analyst of Finance, Economics, Geopolitics and Global Markets.