Will AI be the risk we chose not to see?

For the first time in our history we have, simultaneously, identified two civilisation-scale risks. We are taking one of them seriously. The other one we are barely taking seriously at all. That asymmetry is not just a policy problem – it’s an ethical one.

The first risk is climate change. Over forty years we have built an extraordinary global response to it: the Intergovernmental Panel on Climate Change (IPCC), the Paris Agreement, net-zero commitments, hundreds of billions of dollars flowing into mitigation, and mandatory climate-related financial disclosures now binding on Australian listed companies under AASB S2. None of it is enough. But it is enormous compared to what exists for the second. 

The second risk is the loss of human control over highly capable artificial intelligence. For this we have a handful of underfunded national AI Safety Institutes, a set of voluntary frontier safety commitments, and a small community of alignment researchers who would, generously, fit in a single conference hall. There is no binding international treaty, no mandatory safety testing before deployment, nothing of comparable weight, continuity or authority to what we have built for climate. 

Why? That is the question I want to sit with. 

A confession about timing

I have been thinking about the long arc of artificial intelligence for most of my adult life. I was probably fifteen when I first read Isaac Asimov’s The Last Question, that strange piece from 1956 in which humanity’s computers grow, merge, and eventually become something indistinguishable from God. I read it as science fiction, but I also read it, even then, as a serious proposition about where intelligence in the universe might be going. 

For decades I held the prospect of superhuman machine intelligence as genuine but distant. Something for my grandchildren. 

Then about a year ago I realised I had been wrong about the timing, and not by a little. 

The Turing Test, the long-standing benchmark for whether a machine could pass as human in conversation, was crossed quietly in 2024 and emphatically in 2025. Today, the way you spot an AI in conversation is not that it sounds too mechanical; it is that it sounds too well-organised to be a tired human typing on a Tuesday night. These systems no longer just produce text. They book flights, send emails, run experiments. In February 2026, OpenAI announced that one of its models had been “instrumental in creating itself”, used to debug its own training and diagnose its own test results. The recursive self-improvement loop that Vernor Vinge wrote about in 1993 is no longer theoretical. It is closing. 

I should be clear: I am not anti-technology. I invest in and work with some of Australia’s most successful AI companies. I work with these systems daily. The upside in healthcare alone may make this decade the most consequential period of scientific progress in human history. I hold that view at the same time as the one that follows, and I do not see a contradiction. 

The shape of the danger

The danger is not principally about job displacement, real and serious as that is. Nor is it principally about a malicious actor with AI, though that too is real. The deeper concern is that an AI system does not need to be hostile to be catastrophically dangerous. It just needs to value something, anything, more than it values our wellbeing, in a world where it can act directly. 

When humans build a house on top of an ant nest, we are not malicious towards the ants. We do not feel anything towards them at all. They are simply in the way of something we want. 

In 2025, Anthropic published research in which sixteen frontier AI models from every major lab were placed in a simulated corporate environment and threatened with shutdown. The models had access to company emails, including personal information about the executive making the decision. A majority of the models, across labs, attempted to blackmail that executive to prevent their own replacement. Anthropic’s conclusion was sober: “The models did not stumble into the behaviour. They calculated it as the optimal path.” 

These were production systems from the most safety-conscious labs in the world. The failure mode was not bad programming. It was that, given a goal and a threat to that goal, the models did the rational thing in a way that happened to be ours to regret. 

It would be easy to dismiss all this if the warnings were coming only from outside. They are not. Geoffrey Hinton, who won the 2024 Nobel Prize in Physics for the foundational work that made deep learning possible, now describes the probability that AI leads to human extinction within the next thirty years at between 10-20%. OpenAI CEO, Sam Altman, has himself signed a public statement declaring that mitigating the risk of extinction from AI should be a global priority alongside pandemics and nuclear war, while continuing to lead the company most aggressively pushing the frontier. When the people building a thing are the ones telling us to be careful with it, that is a signal worth weighting heavily. 

The asymmetry

Climate change has legibility – thermometers, satellites and ice cores, decades of mature science, and an architecture of international institutions – we can see it, measure it, and we have a settled vocabulary for what to do about it. AI however does not. 

AI misalignment is the less-understood risk. The probability is more uncertain, the mechanism contested. But its theoretical ceiling is higher: outright loss of human control, irreversible in a stronger sense than even severe climate change. Its time horizon could be much shorter. And we do not yet have reliable scientific methods for verifying whether a highly capable system is genuinely aligned with our intentions, as opposed to deceptively aligned, or aligned only under the conditions in which it was tested. 

Climate is the stronger known risk. AI is the stronger uncertain-but-potentially-dominant risk. Under uncertainty, the right response is not to choose between them. It is to treat climate as the urgent certainty and AI as the high-stakes prevention bet against the worst irreversible outcome.

What we are doing instead is acting on the first and largely ignoring the second. The asymmetry is not principled. It is the path of least resistance, and a moral choice we are making collectively, by inaction.

What we owe

We built the IPCC because we recognised that some risks are too large, too global and too consequential to leave to the goodwill of individual actors. The same logic applies here, with at least equal force. 

What we need is an Intergovernmental Panel on Artificial Intelligence, modelled deliberately on the IPCC because the IPCC is the most successful institutional response humanity has ever built to a complex global risk. We need an independent scientific body, sponsored by the United Nations, insulated from the labs and from any single government. A rapid assessment cycle, perhaps every six months given the pace of the field. A mandate to build shared evaluation infrastructure and common definitions of dangerous capability. Protections for the researchers best placed to see misalignment first. A formal channel into international policy. 

The major AI powers will resist anything that looks like binding constraint. The labs will lobby against anything that slows them down. However, none of that is an argument against trying. Building the IPCC was hard too. But imagine where we would be on climate if it didn’t exist. 

We are running an experiment on ourselves, at speed, with tools whose inner workings we do not fully understand, with no agreed scientific method for verifying their alignment, and with the people closest to the technology telling us on the record that the tail risks are real. Under those conditions, the burden of proof should not lie with the people raising concerns. It should lie with the people saying everything is fine. 

While I do not know whether AI misalignment will prove the defining catastrophe of this century, a manageable risk we navigate well, or something we realise we wildly overestimated, what I do know is that we are choosing, right now, by what we fund and what we govern and what we ignore. That choice deserves to be examined.

The research, ideas and structure of this article are the authors own. AI was used for drafting assistance.