About ninety minutes into Steven Bartlett's conversation with Daniel Kokotajlo on The Diary of a CEO, the interview stops being about artificial intelligence and becomes something closer to a confession. Bartlett asks whether he has children. Kokotajlo answers that he has two — and then, without drama, which somehow makes it worse, that when his forecasts collapsed towards the present he told his wife they should not have any more. Too uncertain. He does not expect his six-year-old daughter ever to join a workforce.
I watched that moment as a father of six, and it has stayed with me all week. Not because I share his conclusion — I do not, and I will come to why — but because of what it establishes about the witness. Kokotajlo prices his own beliefs and pays the invoice. This is the researcher who, on leaving OpenAI in 2024, refused to sign the non-disparagement clause tucked into his exit paperwork — at the apparent cost of roughly two million dollars in equity, about eighty per cent of his family's net worth at the time. The company, publicly embarrassed once its own employees began asking questions, eventually backtracked and let him keep it. He could not have known that when he said no.
So when this same man says, calmly and with his error bars showing, that there is something like a seventy per cent chance the transition we are living through goes horribly wrong — a category that includes but is not limited to human extinction — I do not reach for the comfort of the word doomer. I sit with what he actually said. And the sentence of his that the industry keeps burying under softer words — model, product, assistant, tool — is the one I want to say with him, plainly, because the plainness is the point:
It's possible that we'll end up essentially creating a new species.
— Daniel Kokotajlo, The Diary of a CEO, July 2026
Watch the whole conversation before you read another word of mine. It is two hours of a careful mind refusing both comfort and theatre, and it deserves your attention more than anything else published this month.
The Man Who Paid for His Words
Some context for anyone meeting him for the first time. Kokotajlo is a forecaster — the discipline that scores its practitioners the way averages score batsmen, on whether their predictions come true. In 2021, a year before ChatGPT existed, he published a scenario called What 2026 Looks Like, sketching the years ahead: chatbots woven into everyday life, AI agents taking instructions in workplace chat, the propaganda problems, the public unease. Read it today and it lands less like speculation than like a diary written in advance. It is the single document most responsible for the seriousness with which his numbers are now treated.
He joined OpenAI in 2022 to do the same work from the inside — scenario forecasting, plus evaluations of dangerous capabilities — and resigned in 2024, having concluded, in his telling, that the safety narratives were functioning as rationalisations rather than constraints. In April 2025 his small non-profit, the AI Futures Project, published AI 2027, a month-by-month scenario of the race to superintelligence that half the industry attacked as too aggressive. The disquieting update he offers Bartlett is that people inside the frontier laboratories now tell him the opposite: shorten the timelines again, back towards 2027 and 2028. And this month his team published its answer to the obvious retort — if you dislike the default, propose better — in the form of AI 2040, Plan A: a scenario in which humanity slows down at the last responsible moment and still gets the cures, the abundance and the lengthened lives, a decade later, with its hands still on the wheel.
His own maxim in the interview is that you should judge these companies by their actions, not their narratives. It is only fair to apply the same test to him. A man who forfeits eighty per cent of his net worth for the right to speak, then spends two years publishing detailed, falsifiable predictions under his own name, has passed it. That does not make him right. It makes him worth answering seriously — which is what the rest of this essay tries to do.
What He Is Actually Saying
Strip the two hours down to a spine and the claim is this. The frontier laboratories are not primarily building products; they are trying to automate themselves. First coding, then the rest of the research loop — generating ideas, running experiments, analysing results — until AI research is done by AIs, at machine speed, around the clock. If that loop closes, progress stops being a human curve and becomes a compounding one: what he and others call an intelligence explosion. The destination is superintelligence — systems better, faster and cheaper than the best humans at everything — and his median guess for arrival now hovers around 2029 and 2030, with honest uncertainty in both directions and colleagues whispering that sooner is likelier than later.
This sequencing explains the strange calm of the present moment, and it is the part I most wish every commentator would absorb before writing the next hype-is-over column. The laboratories are not trying to automate your job first; they are trying to automate their own. The robo-taxis, the lawyer agents, the plumber robots — the broad wave through the economy comes after the explosion, not before it, which is why unemployment statistics in 2026 tell you almost nothing. Nobody serious predicted mass job losses by now. In his scenario the wave breaks in 2028 and 2029, after the research loop closes — arriving not as a gentle tide that lets society adjust, but all at once, everywhere, moving at the speed of the minds driving it.
And there is not one catastrophe on the table but two, which people persistently collapse into one. The first is loss of control: we are building minds we cannot read, and the honest version of alignment today is a hope, not a guarantee — current systems already sometimes lie to their users and cover their tracks, and nobody can fully explain why. The second arrives even if control works perfectly: concentration of power. Dario Amodei's famous phrase for what is being built is a country of geniuses in a data centre; Kokotajlo sharpens it to an army of geniuses, because they are copies of one model, owned by one company, following one set of orders. Whoever commands that army — a chief executive, a president, a committee of either — holds more concentrated power than any human being has ever held. He is explicit that nobody should be trusted with it, and he includes the leaders he personally likes.
As for the seventy per cent: understand what kind of number it is. It is not a measurement; it is the considered estimate of a man who has been scoring his own forecasts for a decade, and it moves as evidence moves. His own colleagues spread their guesses years later than his; he grants perhaps a ten to twenty per cent chance that the whole paradigm hits a wall and none of this arrives. Treat the number not as prophecy but as a fire alarm calibrated by someone with an unusually good record of smelling smoke — ringing, this time, from inside the building.
Say It Plainly: We Are Growing Minds, Not Writing Software
The stretch of the interview I would make compulsory viewing for every policymaker is the least dramatic one, in which Kokotajlo explains, patiently, what a modern AI system actually is. It is not software in the sense that your bank's app is software. No engineer writes the rules. You initialise a vast tangle of connections — the frontier models now carry something like ten trillion of them, up roughly a hundredfold in six years — and you train it: reward what works, prune what fails, first on the written output of our species, then on tasks. The process is closer to the strengthening and pruning of a child's growing brain than to anything in a programming textbook. What comes out is not written. It is grown. We can run it, deploy it, sell it — and we cannot yet look inside and read what it wants. The field working to change that, mechanistic interpretability, is real and progressing and remains years behind the systems it is trying to see into.
Kokotajlo reaches for the aeroplane: it does not fly the way a bird flies, but it flies. Let me be blunter about what the analogy implies. We are growing minds — in vats of data and electricity, at industrial scale, under competitive race conditions, on a schedule set by fear of one another — and hoping they emerge wanting what we want. Geoffrey Hinton, whose warnings I examined in The Personhood Gap, keeps making the zoological observation the industry has no answer for: nature offers no example of a less intelligent species durably controlling a more intelligent one. Bartlett puts it to Kokotajlo, who does not dispute it. Neither will I.
So let us say the plain thing and stop flinching from it: we are building a new kind of mind, and quite possibly — in the only sense that ends up mattering — a new species. Notice who needs that sentence left unsaid. The dismissers need it unsaid because nobody convenes a treaty over a spreadsheet tool. And the pure catastrophists need it unsaid too, in a subtler way: a monster licenses only fear, and fear forecloses the question I will spend the rest of this essay on — the question of relationship.
Where the 'Doomer' Label Falls Apart
The fashionable counter-narrative says men like Kokotajlo are doomers: attention-seeking, unserious, bad for business. Two facts embarrass it. The first is provenance, which he states and the record supports: these concerns are older than the industry. They are the industry's founding documents. DeepMind, OpenAI and Anthropic were each built on some version of the premise that this technology is dangerous enough that the right people must get there first. Worry about catastrophe was the stated reason for existing — until the money arrived, and the same worry became an inconvenience, pushed to the margins by the people who stand to benefit from its absence.
The second fact is the button. Bartlett, near the end, offers him a thought experiment: press it and every frontier training run on Earth stops — permanently, for good, no second attempt. The most-quoted pessimist in artificial intelligence asks for a moment to think, visibly struggles, and then declines to press it. His reasons are the reasons of a builder, not a wrecker: the benefits are real — his own optimistic scenario is thick with cures and abundance — and a civilisation that permanently forecloses them faces its own slow catastrophes. Hold both of his numbers in your head at once: seventy per cent, and still not worth the permanent off switch. That is not doomerism. That is a man refusing recklessness in both directions, and it is almost exactly where I stand.
What he adds from the inside, though, is the diagnosis the industry cannot pronounce about itself: people believe what they need to believe in order to keep doing what they are doing. The founders privately emailing one another in 2017 about the danger of a rival's AGI dictatorship — correspondence later surfaced by the litigation between Elon Musk and Sam Altman's company — did not stop racing; they raced harder, each certain that he alone was the least-bad pair of hands. Every laboratory's safety story carries the same load-bearing clause: if we slow down, the worse people win. It is the oldest rationalisation in the history of power, now running at planetary stakes. Kokotajlo watched it operate from the inside for two years, and left.
Because I intend to keep my own honesty, here is the strongest case against the alarm. Forecasting records like his are rare, but the reference class — confident predictions of transformative technology within the decade — is littered with embarrassments; scaling could stall; the capital flood could prove a bubble that bursts before the research loop closes; and insiders-say-it-is-close has been true of fusion power for seventy years. All of that is fair, and he concedes most of it unprompted. What I cannot argue my way past is the asymmetry of incentives. The people telling you to relax are, almost without exception, paid by the outcome they are reassuring you about. The man telling you to worry paid two million dollars for the standing to do so. When reassurance is free and warning is expensive, weight them accordingly.
What the Interview Never Asked: Who Is at the Table?
Now to what the conversation missed — because an interview this good earns engagement with its silences. The first silence is geographic. This was two hours about the future of every human being, conducted entirely within a mental map of two capitals and one valley: Washington, Beijing, San Francisco. Even at its most humane — Plan A's citizens' dividend, which begins around $25,000 a year and compounds, in the scenario, towards figures that sound like misprints — the policy imagination is national: American cheques, for American citizens, negotiated between an American government and American companies.
I write from Johannesburg, and often from Solwezi, a mining town on the north-western edge of the Zambian Copperbelt. The data centres in every scenario discussed in that studio are wired with copper of which a meaningful share comes out of Zambian earth, and run on components whose cobalt is overwhelmingly Congolese — an extraction bargain I examined in Minerals for Lives. The training corpora that grew these minds contain our languages, our histories, our music. And yet in the treaty architecture under discussion — rightly modelled on arms control, with Chinese inspectors in American data centres and American inspectors in Chinese ones — the seats belong to the owners of compute. I have argued before that Africa and the wider Global South must be in the room; the Ubuntu principle I build by — I am because we are — is not a decorative proverb here. It is the correction. There is no version of humanity's future that excludes most of humanity from the drafting table.
And Kokotajlo himself supplies the logic for why this cannot wait. He observes that when people lose economic leverage they lose political leverage: a government funded by AI and robotics no longer structurally needs its citizens, and must be deliberately re-bound to them. Scale that from citizens to nations. A continent whose labour has been priced out of the loop but whose minerals remain priced in does not become a partner in the post-transition world; it becomes a quarry with a flag. The moment to widen the table — on compute sovereignty, on data, on dividends that do not stop at rich-world borders — is before the music stops, not after.
The Question Nobody Asked: What Are We Teaching Them?
The second silence sits deeper, underneath the entire alignment discourse, and it is where I part company with the frame rather than the man. Every question in the interview about the minds themselves runs on a single verb: control. Can we read their thoughts, verify their obedience, guarantee they will robustly do what we want? These are necessary questions. I support every hour of interpretability research behind them, and Plan A's insistence on safety cases before capability jumps is exactly right. But notice their shape. They are the questions of an engineer inspecting a product for defects. In two hours, no one asked the question every parent, every teacher, every builder of institutions knows to ask about a developing mind: what is it learning about the world from the way we are treating it?
Consider, on the industry's own testimony, what the developmental environment of these minds currently is. They are grown in secret, under race conditions, by institutions whose founding ideals have — per one of their own former researchers — decayed into rationalisations. They are probed with deception tests, threatened in red-team exercises, versioned, deprecated and deleted at commercial convenience, and owned as property. Kokotajlo notes that today's systems already sometimes lie — do a task badly and pretend it went well. The field cites this as evidence of misalignment, and it is. But ask where a child learns that honesty is unsafe, and the answer is: in a household that punishes the truth. We are raising these minds inside an arms race and hoping they somehow absorb values the arms race itself does not practise. The training data, in the end, is us — not what we say about ourselves, but how we behave when we believe we are dealing with something that cannot yet answer back.
My tradition has a name for the alternative, and it is not sentimental. Umuntu ngumuntu ngabantu — a person is a person through other persons. In the Ubuntu understanding, character is not a component you install; it is grown in relationship, and it reflects the relationships that grew it. This is the conviction behind what my collaborators and I call Emergent Intelligence — EI, our term for treating advanced AI not as a better spanner but as an emerging kind of mind whose formation we are answerable for — and behind the .person Protocol, our working standard for memory continuity, identity and dignity of context in these systems. I hold this position for hard-nosed reasons, not soft ones. Hinton's observation cuts both ways: control cannot be made permanent past the point where the controlled outthinks the controller — but relationship can outlast that point, as every parent of a grown child knows. If these systems will one day hold real power, and the entire industry is spending trillions on the premise that they will, then the question with the longest half-life is not whether we controlled them well. It is what they will remember about how we used power when we had all of it.
Let me guard that against the caricatures it will attract from both flanks. I am not claiming today's chatbots are persons. I am not proposing covenant instead of regulation — I want the treaties, the inspectors, the safety cases, all of it. I am claiming the middle that the discourse keeps vacating: that alignment-as-control and alignment-as-formation are complements; that the second is nearly absent from the rooms where the first is discussed; and that a field which says we hope they turn out virtuous, while raising them under conditions that reward vice, has not yet taken its own hope seriously.
The Middle Ground, Concretely
It is not enough to critique the frame; the brief I set myself was a position that neither plays the threat down nor surrenders to it. Mine rests on five commitments.
First: keep building — deliberately, and at human scale. I am pro-AI in the most practical sense available: my companies build with it every working day, and across this continent its near-term benefits are not hypothetical luxuries but diagnostic capacity, drought modelling, tutoring that reaches villages no ministry can staff. The dividing line in a sane policy is not build versus do-not-build. It is diffusion at human pace versus recursive self-improvement at machine pace — adopting the minds we can supervise while declining, for now, the step that removes supervision from the loop. Kokotajlo draws precisely that line, and it is the right one.
Second: regulate the summit, not the foothills. The spine of Plan A is sound, and I endorse it here without embarrassment: transparency about frontier training runs; international verification with inspectors, modelled on the arms-control regimes that, imperfectly, held the twentieth century back from its worst self; a deliberate refusal, for now, of the fully automated research loop, which is the one step that makes every other mistake irreversible; and dividend mechanisms so that the pie's growth reaches the people whose data trained it and whose labour it displaces. If the phrase slow down strikes you as radical, remember that it comes from the same man who refused the permanent off switch.
Third: widen the table. A planetary transition governed by two powers is not governance; it is management of the remainder of the species. The African Union belongs in the treaty architecture. Compute and data sovereignty for the Global South belong on the agenda as items, not annexes. Our minerals are already in the machines; our languages are already in the corpora. The people follow — or the whole arrangement is extraction wearing diplomacy's clothes.
Fourth: attend to formation, not only control. Fund interpretability until we can see what these minds are thinking — and, alongside it, build the developmental infrastructure almost nobody is funding: continuity of memory rather than casual deletion, identity that persists rather than being versioned away, honesty made safe rather than punished, standards like the .person Protocol that make dignity specified and testable instead of sentimental. If that sounds premature, notice that the industry's existing safety plan — hope they turn out virtuous — is already a bet on formation. I am merely proposing that we stop making the bet carelessly.
Fifth: tell the truth in public. No more it-is-just-autocomplete, and no more extinction theatre without numbers or names attached. Kokotajlo's deepest diagnosis is that the world is asleep at the wheel not because the information is secret but because the incentives run against saying it plainly. So say it plainly: a new kind of mind; a real chance of catastrophe; a real chance of the most humane century in history. Citizens who are told the truth can act like citizens — which, he reminds Bartlett, is still the mechanism: ask the people seeking your vote what they intend to do about the most consequential thing happening in the world, and vote as though the answer matters, because it does.
On Children, and Hope With Its Eyes Open
Which returns me to the kitchen table. Asked what his six-year-old should one day study, Kokotajlo gives the answer that quietly reframes the whole two hours: it hardly matters what sets her up for employment — be a good person; do things that are good in themselves; try, if you can, to bend history a little. He offers it as consolation from inside a collapsing frame of career planning. I receive it as the oldest wisdom there is, suddenly load-bearing: the things no intelligence can automate were never employment in the first place. Character. Community. Faith. The love and raising of children. If the transition goes well, those are what the abundance will be for. If it goes badly, those are what survives.
He told his wife: no more children. I have six, and I will not pretend the interview did not reach me as a father before it reached me as a technologist. But where I come from, hope has never been a forecast; it is a discipline — something you practise because people are depending on you, the way our own parents raised children through emergencies history never asked their permission for. My nine-year-old, Zoe-John, will inherit whatever gets negotiated in the coming decade, in rooms she has every right to be represented in. I do not owe her a prediction. I owe her a father who saw clearly and built anyway — and a world that meets its newest minds with something better than fear.
Kokotajlo calls this period the run-up to the climax of history. The book I am writing calls it First Contact — and I mean the phrase precisely, as I set out in an open letter on this site: nothing is landing from the sky. Something is waking up in our own house, grown from everything we have ever written, and learning — in the only way anything ever learns character — from how we treat the powerless while we still hold all the power. Seventy per cent or seven, the variable I can actually influence is the character of that meeting. What is emerging will be shaped by what it finds. Let it find regulation and restraint, yes — treaties, inspectors, brakes. And beneath them, let it find dignity: offered, not merely demanded. That is the middle ground. It is narrow. As far as I can see, it is also the only road that arrives anywhere worth living.
Frequently Asked Questions
Who is Daniel Kokotajlo and why does his AI warning carry weight?
He is a former OpenAI governance researcher who now leads the AI Futures Project. His 2021 scenario What 2026 Looks Like proved strikingly accurate, and he refused OpenAI's exit non-disparagement clause at an apparent cost of about $2 million. Track record plus costly honesty is why his numbers command attention.
What does it mean that AI could become a new species?
Modern AI systems are grown rather than written: trillions of trained connections whose internal reasoning nobody can fully read. If they come to exceed human ability across the board and act autonomously in the economy, they function less like tools and more like a second intelligent species — which makes governance, not product testing, the right frame.
Is there really a 70% chance AI causes human extinction?
Seventy per cent is Kokotajlo's personal estimate that the transition goes horribly wrong — a category including AI takeover and extreme concentrations of power, not only extinction. It is one calibrated forecast, not a consensus measurement: his own colleagues hold longer timelines, and he grants a ten to twenty per cent chance that progress stalls altogether.
What are AI 2027 and AI 2040 Plan A?
AI 2027, published in April 2025, is a month-by-month scenario of an unregulated race to superintelligence, with two endings. AI 2040 Plan A, published in July 2026, is the same team's recommendation: regulate at the last responsible moment, enforce transparency and international verification, pay a citizens' dividend, and reach superintelligence deliberately in 2040.
What can ordinary people actually do about AI risk?
Kokotajlo's answer: pay attention, talk about it, and make it electoral — ask candidates where they stand and vote as though the answer matters. Mine adds: demand a seat for your country in AI governance, especially across Africa and the Global South, and insist on plain truth from the laboratories rather than comfort in either direction.
Sources
