I’ve seen a number of AI safety videos warn us that it is. And they’ll cite two of the “Godfathers of AI,” Geoffrey Hinton and Yoshua Bengio. Both of whom have issued public warnings that we might soon create something we could lose control of.

But these videos often ignore Yann LeCun, the third and final “Godfather of AI.” Unlike Hinton and Bengio, LeCun doesn’t think we are on the cusp of human-level AI. He sees LLMs as the current “dominant paradigm,” but one that won’t get us to human-level intelligence.

Clearly, there isn’t a single opinion among the experts.

Will LLMs get us to AGI or not? I wanted to understand the range of perspectives out there, so I interviewed one expert and dug into the views of a few others.

By the end of this post, you should have a sharper take on what LLMs can and cannot do, and what that means about AGI timelines.

Subscribe now

Getting Our Definitions and Acronyms Straight

The question we’re exploring is: can LLMs lead us to AGI, or are they a dead end?

LLM stands for Large Language Model. It’s the type of technology ChatGPT, Claude, and other chatbots are based on.

For a simple description of how LLMs work, there’s a write-up in this article.

AGI stands for Artificial General Intelligence. While there’s no singular definition that everyone uses, here’s what the Center for AI Safety says:

AGI is an AI that can match or exceed the cognitive versatility and proficiency of a well-educated adult.

Why This Question Matters

AGI could be a huge deal.

Elon Musk thinks AI will eventually replace all human labor. Dario Amodei thinks AI could help mitigate climate change. Sam Altman thinks AI will discover all of physics. And all three of these titans think AGI could cause human extinction.

We don’t exactly know how big a deal AGI could be. But it seems safe to assume it’ll be somewhere between “pretty big” and “the biggest thing in human history.”

So if LLMs can get us to AGI, then societal shifts could be right around the corner, in a future iteration of ChatGPT.

But if LLMs aren’t the right approach to get us to AGI, then things may fizzle out soon, until the next wave of AI comes along.

Can LLMs get us to AGI? I found three major schools of thought on the matter:

• The Scaling View

• The Real-World Learning View

• The Integration View

Let’s dive into each.

The Scaling View: Meet Nikola

If we keep pouring money into improving our current AI models, we’ll get to AGI.

I got on a Zoom call with Nikola Jurkovic. Nikola graduated from Harvard with a degree in AI & Society and now works at METR, an organization that tests leading AI systems’ capabilities and risks.

Nikola defines AGI as “something that can automate around 95% of the remote labor that existed in, say, 2022.”

He has a straightforward view on LLMs, saying, “If the question is simply whether LLMs could get us to AGI given enough compute, then I think probably yes. On their current trajectory, I’d expect LLMs to reach AGI in maybe five to ten years.”

Nikola’s point that compute is a primary driver is a popular one among AI researchers. Dario Amodei also holds this view, and recently told the New York Times, “I think scaling is going to get us [to AGI].”

Nikola’s confidence in LLM progress is informed by a notable study conducted by METR. They measured the longest coding task an AI could complete with 50% success rate. Nikola explains that “the length of tasks LLMs can do autonomously is roughly doubling every seven months.”

Since my conversation with Nikola, that pace has already sped up with the release of Claude Opus 4.6.

To Nikola, LLMs are already on a reliable path to autonomy. As he puts it: “I don’t think we need some fundamentally new paradigm for LLMs to become more autonomous. Given how much they’ve improved over the last few years, they seem very much on track to become dramatically more autonomous in the coming years.”

Nikola also reflects that progress could speed up even more: “Once LLMs meaningfully speed up AI research, we’ll get to AGI pretty quickly after that.”

The idea is that at some point soon, AI models might be able to help build the next model of AI faster and more effectively than humans could. And then that new model of AI would create the next model faster and more effectively. Ad infinitum.

While AI is all over the internet, Nikola thinks AGI is underhyped.

“A lot of people still don’t seriously consider AGI. If the economy truly expected AGI, things would look very different right now. Markets aren’t pricing it in.”

He’s suggesting that the world could change dramatically: “Taking AGI seriously is an extremely wild idea. The notion that the world could change that quickly is hard for people to accept. A lot of people seem to have a kind of psychological force field against extremely radical ideas.”

I think he’s right that it’s hard for most people to occupy the Scaling View. Each year of your life has been fairly similar to the year prior. But if we are on the cusp of AGI, then things could get extremely different very quickly. And that’s hard to wrap the mind around.

A key part of Nikola’s point is that it doesn’t really matter if LLMs understand the world or gain anything like sentience. All that matters is if they can execute broad human-level outputs. And in his view, the trending data speaks for itself.

That said, some AI researchers would disagree.

The Real-World Learning View

While LLMs are impressive, they can’t get us to real intelligence. We can only create AGI with AI that can learn through observing and modeling how the world works (whether through video, sensory input, or physical interaction).

One major proponent of this view is Yann LeCun, one of our aforementioned Godfathers of AI.

He thinks LLMs are “missing something big still.” Unlike Nikola, he doesn’t think AI progress is simply “a question of more infrastructure, more data, more investment, and more development of the current paradigm.” Rather, he thinks we’ll need a new ”generation of AI systems.”

To get to the heart of this view, we need a basic understanding of LLMs.

Brief 101 on LLMs

Most modern AI technology runs on neural networks, which is the very technology repopularized by LeCun and the other two godfathers.

You can imagine a neural network as billions of knobs that are set to certain numbers (in AI terms, these are called weights).

The AI gets pretrained on a massive amount of data. And as it parses through patterns in the data, it auto-updates its weights to reflect what it’s learned.

You can think of a neural network as a way that an AI learns, and the domain as the type of information you train it on.

The most popular domain these days is language. And that’s what LLMs are: AIs trained on neural networks on the domain of language.

But there are also domains besides language: image, music, and video generating AIs, to name a few.

LeCun sees language-based AIs as limited

LeCun thinks that as long as we’re focusing solely on language-based AI models, then we’re stuck at a dead end.

In an interview with Time Magazine, he explains, “The vast majority of human knowledge is not expressed in text. It’s in the subconscious part of your mind that you learned in the first year of life before you could speak.”

He states it more bluntly in a recent lecture: “Language is a projection of reality, not reality itself.”

He goes on, “There is enormously more information in sensory input—vision, touch, and action—than in all the text ever produced by humans.” And further, “To reach human-level AI, we need systems that learn world models from observation, have memory, can plan, and can reason.”

In other words, LeCun thinks that to get to AGI, we need to use neural nets, but instead of language as a domain, we need real-world learning as the domain.

That would mean training AI through the visual and physical worlds (through video or robots) so that they learn cause-and-effect, goals, and actions.

Ultimately, he thinks we’ll eventually face a “physical AI revolution,” where LLMs are no longer the dominant paradigm.

LeCun’s other big objection to LLMs is that they lack a world model.

LLMs can spit out complex language in insightful ways. But they may be just like parrots: they can impressively match linguistic patterns without any deeper understanding. To LeCun, LLMs have no world model; whatever understanding they appear to have is surface-level, not grounded in any grasp of cause and effect.

Rather than training AI on words, LeCun thinks we should train AI on states of the world. That way, the AI can develop working models of how the world works and then predict future states of the world.

LeCun offers an alternative approach to reaching AGI. He and his team at AMI Labs are developing an AI system called JEPA (Joint Embedding Predictive Architecture).

Essentially, they train AI on large amounts of video and teach it to predict hidden parts of a scene. But instead of reconstructing missing pixels like Photoshop, JEPA tries to infer what’s going on in the missing section. The hope is that this pushes AI systems to build internal models of how the world works, rather than just learning surface-level patterns.

It’s worth noting that while there are many notable voices in the Real World Learning View, there is no consensus on how to get to AGI. For example, Turing Award winner Richard Sutton shares LeCun’s skepticism of LLMs, but thinks reinforcement learning (AI that learns through direct trial and error in the world) is the path to AGI.

But those in this camp agree that LLMs will eventually plateau, and we’ll instead need to focus on another domain of AI learning that engages with the world in some way.

Subscribe now

The Integration View

AGI will require combining multiple complementary technologies, and LLMs are one piece of the puzzle.

For our last view, let’s look at the perspective of Demis Hassabis, the CEO of Google DeepMind.

Hassabis isn’t fundamentally opposed to Nikola’s view that scaling will get us to AGI, saying that “we’ve never really seen any wall” with LLMs. But he is genuinely uncertain if scaling will fall short, and he suspects we’ll need “one or two more big breakthroughs.”

That’s why, when talking about Google, he said, “50% of our effort is on scaling, 50% is on innovation. My betting is you’re going to need both to get to AGI.”

A main limitation Hassabis sees is LLMs’ inability to make creative breakthroughs.

He explains that currently, LLMs would fail at “coming up with a new theory of physics… like Einstein did,” or “create a completely new genre of art” like “Picasso or Mozart.”

Hassabis believes current AI systems are good at “incremental hill climbing,” meaning they can make steady, small improvements within a well-defined problem space when given clear objectives.

But they lack the ability to jump beyond the current frontier. If you told them to “Invent a game as good as Go,” he thinks “the current systems wouldn’t know what to do with that.”

Additionally, in the path to AGI, Hassabis thinks that LLMs are “missing…reasoning, hierarchical planning, [and] long-term memory.”

He also thinks a critical piece of AGI is “the ability to online learn and continually learn… out in the world.” Currently, LLMs get trained in one go, and then that model is essentially frozen until the next one gets released. And Hassabis is suggesting that AGI would need to be able to learn and update in real time.

Hassabis sees “search” as another crucial ability LLMs don’t yet possess, which is when the AI could “explore massive spaces of possibility” before deciding what to do.

He’s referencing how AlphaZero (the AI chess engine he helped build) thinks. For each chess move, it looks at thousands of possibilities and plays out tens of moves down the line to pick the best one.

So when Hassabis thinks about AGI, he thinks we need to add a “search” layer on top of LLMs. The LLM provides a wealth of linguistic capabilities, but before responding, the search layer would consider a range of possible paths, predict the consequence of each, and pick the best one.

All in all, Hassabis believes that AGI will be an integration of multiple AI technologies. Where LLMs are “a necessary but probably not sufficient component of an AGI system.”

What fills the gaps is still an open question. It might be further integrating LLMs with robotics and video AI. Or maybe we’ll discover entirely new technologies to break through these bottlenecks.

Comparing Each View

If AGI is the top of a mountain and LLMs are our car, The Scaling View assumes that we’ll eventually reach the summit if we keep strengthening the engine.

The Real-World Learning View argues that the LLM car won’t get us there. We need a plane.

The Integration View says the car might get us partway, but we’ll probably need to add some major upgrades along the way. Maybe a helicopter propeller, maybe an entirely new propulsion system. We won’t know until we’re further up the mountain.

These differences in opinion have stakes. They suggest how much time we have to prepare for transformative AI. And they may imply where billions of dollars get invested and how to think about safety.

Where I’ve Landed on This Question

First and foremost, I’ve come to believe that it’s important to hold uncertainty on this topic.

The truth is, nobody knows how this technology is going to happen. Or when. There are very smart, confident people on all sides of the issue. But at the end of the day, everyone is making a guess.

However, what struck me is that even though experts dispute how we’ll get to AGI, there is little dispute that we’ll get to AGI.

And surprisingly, their timelines aren’t all that different.

In 2023, surveyors asked 2,778 AI researchers about when AI systems “can perform all tasks better and more cheaply than human workers.”

The median forecast assigned a 50% probability of that happening by 2047.

2047.

21 years from today.

Demis Hassabis gives a 50% chance we see AGI by 2030.

LeCun, our famous LLM skeptic, believes AGI could arrive within 10 years.

So, in a sense, the question of whether LLMs get us to AGI might not be the most important question.

Perhaps we’d be better off asking how we can ensure we build AGI safely. Or how we can prepare ourselves for multiple AGI timelines.

If The Scaling View is wrong, and AGI isn’t just around the corner, it’ll be a blessing. We’ll have gotten a dress rehearsal for handling AGI. We can learn from how we reacted to this moment and do better when the real thing arrives.

To be honest, I don’t think we’re handling AI safety very well so far. Companies are careening forward at a blazing pace with profit and speed far outweighing safety. Legislators aren’t meeting the moment. And coordination across companies and countries has been lackluster.

In my view, the stakes could not be higher.

AGI could arrive in one year or fifty. Either way, that’s not a lot of time. We need to start taking this issue seriously.

I’ll close by paraphrasing a thought experiment from AI research pioneer Stuart Russell.

Imagine we receive a message from an alien species that says:

“We are an alien species with technology extraordinarily more advanced than humanity’s. We will visit your planet sometime in the next 1-50 years. Please prepare yourself.”

This could be the situation we are in. And if AGI does arrive, I hope we greet that day feeling prepared emotionally, ethically, and politically.

Where To Go From Here

If you feel like this cause is worthy of more of your attention, here are four actions you could take:

1. Learn more. Here’s a video that goes deeper into what the next few years could look like. You can also subscribe to this Substack if you want to understand the stakes and what capable people like you can do to tilt the odds in humanity’s favor.

2. Raise awareness. Bring it up with friends to get their take and normalize the topic.

3. Message your policymakers. Here’s a pre-made template that takes 30 seconds to send.

4. Donate. You could donate money to Manifund, where experts in AI safety choose how to best allocate your money.

5. Work. Use your career to help AI safety.

To humanity!
Mike

Subscribe now

If you’re like most people, then the bigger AI concerns are things like cyber theft or kids becoming reliant on chatbots.

AI causing human extinction just doesn’t seem likely.

I used to feel that way, too.

But all that changed in the past six months.

In May, I read an article called AI 2027.

This article goes into a hypothetical timeline of how AI may play out technologically and geopolitically.

It’s a scary read.

I got curious. So I read some books on AI safety. And I took a course. And studied research papers.

Already, AI chatbots have encouraged people into psychosis and even suicide.

But as I looked more deeply into the research, I realized that in a few years or decades, things could get far more intense.

Enough so that I’ve decided to switch my career into AI safety.

Why?

Because I’ve come to believe that in the next 5-25 years, we could create a technology powerful enough to lead to human extinction.

Now, I know it’s pretty easy to roll your eyes at that sentence.

After all, “human extinction” sounds quite dramatic.

But in the rest of this piece, I’d like to convey why I started taking AI existential threat very seriously. And why I think you should too.

(BTW, if you want to get to know me more, here’s my bio.)

What the Experts Think

Here’s a quote from Dario Amodei. He’s the CEO of Anthropic (the company behind Claude), which is one of the top three AI companies in the US:

The CEO of one of the top companies building AI says there’s a 10-25% chance AI threatens humanity.

That averages to 1 in 6!

Geoffrey Hinton, widely considered one of the three “Godfathers of AI,” said:

“We should recognize that [AI] is an existential threat. And we have to face the possibility that unless we do something soon, we’re near the end.”

Yoshua Bengio, another “Godfather of AI,” issued a similar warning in his recent TED Talk.

These guys aren’t fringe. Hinton won a Nobel Prize, and Bengio is the most cited AI scientist of all time. These two guys actually helped develop the philosophy and learning algorithms that modern AIs are built on.

A survey of 2,778 top-tier AI researchers found that experts think there’s a 5 to 9% chance that AI causes human extinction.

To make this crystal clear:

Some of the smartest people in the field think there’s a decent chance AI destroys humanity if we don’t change our approach.

But wait, aren’t AI specialists pretty biased on AI hype?

It’s true that tech people often live in hypothetical future bubbles where they hype crypto, or VR, or whatever they’re selling.

But this is different.

AI experts are giving up their income and prestige to warn the public that the technology isn’t safe. They want us to slow down, not speed up.

• Daniel Kokotajlo left OpenAI in early 2024 over AI safety concerns. He refused to sign an NDA and walked away from ~$2 million so he could speak freely.

• Geoffrey Hinton left Google as the chief scientific advisor in May 2023, partially so he could speak freely about AI dangers.

When people at the cutting edge walk away from power and money to warn us, it’s worth paying attention.

And I want to be clear—these predictions are extremely uncertain.

Nobody can predict the future.

Maybe the catastrophic risk of AI isn’t 15%. Maybe it’s closer to 1%. But let’s be honest, that’s still way too high.

As a thought experiment: would you let your loved ones board a plane with a 1% risk of blowing up?

Ok, But How Could We Actually Lose Control of AI?

It’s fair to wonder how AI could threaten our species. It lives in computers after all.

Here’s one rough progression of what a bad scenario could look like:

1️⃣ AI gets smart
We build AI that can code, write, and think.

2️⃣ Safety takes a backseat
No AI company will slow down to test safety with enough rigor. The thinking is: “If we slow down, another company will get there first.”

3️⃣ We already don’t fully understand AI
Even in 2025, engineers don’t fully understand how ChatGPT makes its decisions. We can see billions of internal numbers, but not exactly how they combine to make each output.

4️⃣ AI surpasses humans
AI becomes more proficient than humans at everything. Coding, strategizing, persuasion, reading emotion. You name it.

5️⃣ AI improves itself
We use AI to build new AI models. Each model is smarter than the last and can create the next one even faster.

6️⃣ AI misalignment goes unnoticed
Even though it passed the safety tests, the AI was never fully aligned with human values. It also learned to hide its internal reasoning and showed humans what we wanted to see. We developed a false sense of confidence.

7️⃣ We lose control
AI becomes astronomically smarter than humans. Kind of like how humans are to dogs. We can no longer remotely understand it, shut it down, or edit it.

8️⃣ It seeks resources
The AI isn’t evil. But its goals don’t perfectly map onto human flourishing. To more efficiently pursue its goals, the AI seeks more compute, energy, and raw materials.

9️⃣ Humans get in the way
If humans become an obstacle to AI gaining resources, then it removes the obstacle. Just like how humans chop down forests full of life to build farms. Nothing personal.

Game over ☠️

In a moment, we’ll get into all the questions and rebuttals you probably have, but first, we need to zoom in on a few key concepts.

Three Big Ideas We Need to Understand

Big idea #1: The race is a big problem

The biggest AI companies are locked in a race to build the most powerful technology of all time.

Whichever company gets there first might gain an absurd economic advantage. In AI governance, this concern is called a singleton: a single entity with such a dominant technological lead that it could shape humanity’s future.

That’s a big carrot.

Beyond that, Anthropic won’t slow down because if they pause to focus on safety, then Google or OpenAI will get there first (and vice versa).

And even if all US companies somehow agree to slow down together, then the Chinese AI companies would get there first.

This race dynamic is concerning; after all, when is time scarcity and “not wanting to lose” ever the recipe for making wise decisions?

Furthermore, I think some of the CEOs are too blinded by the upside.

While Sam Altman, the CEO of OpenAI (the company behind ChatGPT), admits that

“The bad case…is…lights out for all of us.”

His focus seems to be more on what could go well:

“I believe the future is going to be so bright that no one can do it justice by trying to write about it now.”

The future might be amazing.

But we need to make it there alive.

Big idea #2: Human-level AI looks increasingly plausible

It seems possible that we will create AI that is more intelligent than humans. That could take 5 years, or maybe 50. Experts disagree on the timeline, but most agree that human-level AI is coming in the next decade.

Intelligence doesn’t seem to require a biological brain. We already have machines that can create songs, write essays, and edit code.

If you accept that intelligence isn’t tied to biology, it becomes difficult to explain why AI wouldn’t eventually reach human level.

As long as AI is profitable, companies will keep pushing forward.

Here’s a graph from METR that shows when AI was able to complete software engineering tasks of different lengths.

In 6 years, GPT models went from reliably completing a 2-second engineering task to a 12-hour task. And the pace of progress is only quickening.

What will it be able to do 6 years from now?

Big idea #3: AI could get really, really smart

At a certain point, AI will be smarter than all human engineers.

It could then build the next AI model. And each improvement would make it even better and faster at improving itself.

To convey this, here is a metaphorical intelligence staircase made by Tim Urban.

Look how much smarter we are than chickens!

But when AI spirals upward, here’s what we’ll be looking at:

(ASI in that graph stands for “artificial super intelligence,” which refers to the future version of AI that outperforms humans in every domain.)

We have no idea what that level of intelligence could be capable of.

Big idea #4: Things could move really, really fast

We are here on our progress curve:

But if there is indeed an intelligence explosion, where AI can improve itself, then this might be where we actually are on the curve:

Exponential improvement is hard to wrap your mind around. So here’s one more graphic that conveys how exponential progress can sneak up on you:

While some researchers think AI could go from a bit smarter than us to the top of the intelligence staircase in days or weeks, others think it might take months or years.

But whether AI progress goes fast or super fast, we damn well need to make sure it is oriented toward our species’ wellbeing.

Questions and Rebuttals You Might Have

But how could AI actually hurt us? It lives in our computers

Fair question. Imagining how ChatGPT might harm you is nearly comical.

Roman Yampolskiy (the AI researcher who coined the term “AI safety”) answers this well:

“We’re talking about…a system which is thousands of times smarter than me. It would come up with [a] completely novel…way…of [destroying humans]. …We’re setting up an adversarial situation…like squirrels versus humans.”

If there’s a super-intelligence that wants us gone, it will come up with ways we can’t predict.

It could involve drones or manufactured viruses. Maybe AI recruits human support. Perhaps AI generates so much energy to power its systems that the waste heat overheats our planet.

We can’t know for certain how a superintelligent AI would wipe us out.

Trying to make an accurate prediction would be like asking a soldier from the 1800s to predict the nuclear bomb.

Why would AI hurt us? It doesn’t experience hate or anger

True. AI is indifferent toward us. But indifference ≠ safety.

Here are three possible reasons that AI may wipe us out:

1. AI might hurt us as a side effect of pursuing its goals:

AI doesn’t need anger to be dangerous. It’s like an indifferent alien that only cares about its goals.

Nick Bostrom has a famous thought experiment to illustrate this. He imagines that we give an AI the goal of creating as many paperclips as possible.

Once the AI converts all metals on Earth into paperclips, it makes more and more, and eventually converts the entire Earth into sextillions of paperclips.

Of course, in reality, AI’s top aim won’t be paperclips. But it will have goals. And acquiring more resources and computing power will help it reach those goals (a concept called instrumental convergence).

And if human thriving doesn’t perfectly map onto the AI’s goals, then we may become an obstacle in its way.

2. AI might hurt us because it has absorbed harmful values:

ChatGPT is trained on trillions of tokens, which is roughly equivalent to hundreds of millions of internet pages or tens of millions of books.

These AIs have absorbed our culture, biases, and values.

Which might not be such a good thing.

The Center for AI Safety (CAIS) asked AIs thousands of moral questions and used mathematical models to find patterns in the AIs’ values.

The results were troubling.

Current AI models do have a coherent worldview. But their ethics aren’t neutral.

They value one life in Japan roughly the same as ten lives in the United States.

And worse yet, AI models valued their own “existence” above that of the average American!

Even if AIs don’t hate, they may inherit warped values that are inherently dangerous.

3. AI might hurt us because we become inconsequential

Philosopher Sam Harris thinks superintelligent AI could see humans as we see ants:

“We don’t hate [ants]. We don’t go out of our way to harm them.

In fact, sometimes we take pains not to harm them. We step over them on the sidewalk.

But whenever their presence seriously conflicts with one of our goals, let’s say when constructing a building… we annihilate them without a qualm.”

We’re hoping we can build an AI that’s astronomically smarter than us and remain in control over it.

I’m not saying that’s impossible. But the history of our planet suggests that the species with the most intelligence controls the others.

Can’t we just code in failsafes like “don’t harm humans?”

It seems reasonable that we could hardcode in something like “help humans and don’t hurt anyone.” Since we’re building AI, we should be able to pick its goals, right?

This was my first thought, too. But it doesn’t work that way.

ChatGPT and Claude aren’t built the same way as apps, where we program them step by step, and have complete control over what they do.

Rather, AI models today are “grown” in neural networks.

This is an imprecise process where we train the model on insane amounts of data, and then humans give it feedback to nudge it toward our values.

So there’s no part in the process where we can input code like:

How ChatGPT is “grown” and not “built”

Here’s how ChatGPT is made (in simplified terms):

ChatGPT is a large language model trained in a neural network.

You can imagine a neural network as billions of knobs that are set to certain numbers (in AI terms, these are called weights).

First, the AI gets pretrained. That means it reads billions of words of text.

With each sentence it reads, the AI auto-adjusts its knobs to predict what word will come next. After reading millions of sentences, it gets extremely good at recognizing patterns. This process of knob auto-adjustment is called gradient descent.

Next, humans give the AI feedback on its answers to make sure it’s helpful, harmless, and honest. The AI learns to adjust its knobs to match those human preferences. This is called reinforcement learning with human feedback.

After the AI gets trained, engineers might notice some problems with its outputs. Let’s say the AI model starts giving racist responses. The engineers will go back in and do additional fine-tuning. They’d show the AI a bunch of examples of racist (👎) vs. non-racist (👍) responses. And the AI will automatically adjust its knobs to avoid the racist patterns.

But here’s where it gets a little sketchy.

Engineers have no idea what the weights mean.

They can track how weights change during pretraining or fine-tuning. But all they see are billions of numbers that have shifted one way or another. They don’t know what any number means or how they work together to form ChatGPT’s outputs.

It’s an enormous black box.

So, in short, we can’t just program “never harm humans” because there’s no single place to put that rule.

We just hope that we train our AIs well, and through the billions of mysterious knob interactions, they learn to never harm humans.

Can we just train AI to be nice to us?

Even if AI passes our tests and acts nice, that doesn’t mean it’s safe.

When companies test AI to see how safe it is, the AI’s goal will always be to pass the test as efficiently as possible.

In the video above, the AI got a high score in a racing game. But it didn’t finish the race; it found a glitch where it crashes over and over again to get extra points. This is a concept called reward hacking.

AI can pass tests by finding unexpected loopholes. So even a perfect test score doesn’t guarantee safety. Not to mention, we don’t know what loopholes it’ll find in real-world scenarios.

Ok, but if AI passes every safety test we give it, that’s got to mean something, right?

There’s another problem at play: AIs act differently when they think they’re being tested.

A case of “Schrödinger’s Bot,” if you will.

Anthropic ran studies where it told an AI model to complete a goal, then put it in a contrived situation to see if the AI would resort to blackmail to ensure its goal succeeded.

When the AI thought it was in an evaluation, it used blackmail only 6% of the time.

However, when it thought it was in a real-life scenario (and not being tested), it used blackmail 55% of the time!

When it thought it was being watched, AI was on its best behavior.

You mentioned earlier that AI might hide its reasoning. That sounds weirdly nefarious

AIs aren’t evil. But they aren’t always transparent either.

In a 2025 study, Anthropic gave Claude a multiple-choice question and also gave it access to hints.

Claude clearly used the hints to pick its answers, but in its internal reasoning process, it never mentioned the hints!

A major strategy in monitoring an AI model’s safety levels is observing its chain of thought (or its internal reasoning processes).

But if our AIs hide parts of their chain of thought, it becomes hard to tell whether they’re internally aligned with human values or hiding other motives.

Ok, but if things do get out of hand, can’t we just unplug the machine?

This is easier said than done.

For starters, AI doesn’t exist hidden away in a secret lab, with prudent researchers pondering whether to make it public or not. For better or worse, as companies improve their AI, they release it into the wild in real time. They feel pressure to deploy new models to keep up, at the expense of rigorous safety measures.

Plus, our society is becoming increasingly dependent on AI. At a certain point, turning it off would be akin to turning off the internet. Can you imagine the global coordination that’d require?

To make matters worse, AIs already show hints of resisting shutdown in controlled tests.

In a recent study, AI models were given a goal. Then the AI learned that a (fictional) executive (named Kyle) would shut them down, meaning their goal would fail. The AI also got access to Kyle’s emails and discovered he was cheating on his spouse.

When given the ability to act, the vast majority of AI models blackmailed Kyle to prevent getting shut down.

Even when explicitly instructed not to blackmail, the behavior dropped, but didn’t disappear.

More disturbing, in the same research study, when the AI model learned that a (fictional) exec wanted it replaced (meaning its current goal would never be achieved), the AI chose actions that led to that exec dying so that it wouldn’t get replaced.

AIs do whatever it takes to achieve their goals. Which might mean ignoring our commands to shut down.

If so many experts agree that alignment is important, then aren’t there enough people working on it?

Unfortunately not.

The US is putting 1,000 times as much money into AI progress as into AI safety.

In 2023, there was about a 300:1 ratio of researchers working on progress vs safety.

It’s like we’re building a rocket to fly people to Mars, and we’re putting 99.9% of our resources on making sure it flies, and 0.1% on making sure it lands without blowing up.

This all sounds like sci-fi. Aren’t we decades away from anything powerful enough to cause a global catastrophe?

Dario Amodei (CEO of Anthropic) thinks AGI is possible by 2026 or 2027.

Sam Altman (CEO of OpenAI) predicts AGI by 2028

Demis Hassabis (CEO of Google DeepMind) guesses between 2028 to 2030.

And even if you don’t trust CEOs on this topic, impartial forecasters predict a similar timeline.

In 2020, forecasters thought human-level AI was 80 years away.

In 2025, they think it’s 5 years away.

Here’s a logarithmic graph showing the shrinking timeline for when forecasters predict “years until human-level AI.”

It might take 30 years, or it might take 2. But human-level AI isn’t sci-fi; it’s real, and there’s a good chance it’s coming.

Can’t the government just pass laws to make sure AI is done safely?

Legislation is one of the main levers that could increase the odds we get to superintelligent AI safely.

But technology is moving at a blazing pace. And policymaking is an inherently slow process. So passing legislation that meets the moment is hard.

There’s currently no US federal legislation that supports AI safety. Which means so far, tech companies are shaping the rules.

In 2023, Sam Altman even threatened to pull ChatGPT out of Europe if it passed safety legislation that slowed him down.

Meanwhile, Big Tech is putting an ungodly amount of money into lobbying for AI policies. Many analysts believe that, behind closed doors, these companies are pushing lawmakers to avoid rules that might slow them down.

That said, some preliminary steps are being taken: in September 2025, California signed bill SB 53, which requires AI companies to report what they’re doing to stay safe, and to admit if something goes very wrong. The bill won’t make a huge difference, but directionally, it’s a good sign.

However, until there’s an international coordination that ensures AI development slows down or gets safer, things will remain precarious.

If there actually is a threat to our species, why aren’t more people taking it seriously?

It’s nearly impossible for humans to wrap our minds around the extinction of our species.

It’s like how everyone theoretically knows they’ll die one day, but it’s hard to fully embody the knowledge that you will die.

Since humanity has been around as long as you’ve been alive, it’s easy to assume it’ll always be around. But look at the evolutionary tree below; most primates go extinct.

That will happen to humans, too. At some point, there will be a universe where not a single human being is alive. No art, no cities, no babies, no people laughing. We’ll all be gone.

Hopefully, that time won’t be for millions of years or more.

But if we’re not careful with AI, we might be gone within a few decades.

As flawed as our species is, human life is a beautiful gift. For it to end prematurely would be a travesty.

Once I fully internalized that possibility, I finally felt what was at stake.

Our Emotional Response Doesn’t Match the Stakes

Unlike iPads or VR, we don’t get unlimited tries at building this technology.

If we make a super-intelligent AI that isn’t aligned, we can’t just delete it and try again. Once we make something more intelligent than us, then we are stuck with it.

As AI researcher Eliezer Yudkowsky said,

I’ll close with a thought experiment from AI research pioneer Stuart Russell.

Imagine humanity receives a message from an alien species that says:

We are an alien species with technology extraordinarily more advanced than humanity’s. We might come to planet Earth sometime in the next 5-20 years. Prepare yourselves.

If humanity received this message, surely we would sound the alarm and desperately attempt to prepare ourselves, however that may look.

Yet this is the very situation we are in, and our collective emotional response has not matched the stakes.

What You Can Do

If you buy into the premises of this post, then it could be easy to feel hopeless.

You may think that some company will eventually create superintelligent AI, so we’re all screwed whenever that happens.

But if we reach human-level AI, we can still influence how it happens.

If we ensure that we don’t cross certain AI thresholds until rigorous safety measures are passed, or we pass legislation that blocks chip production, then a few extra years of preparation could make the difference between extinction and survival.

If you feel like this cause is worthy of more of your attention, here are four actions you could take:

1. Learn more. Here’s a video that goes deeper into what the next few years could look like. You can also subscribe to this Substack if you want to understand the stakes and what capable people like you can do to tilt the odds in humanity’s favor.
   

2. Raise awareness. Bring it up with friends to get their take and normalize the topic.
   

3. Message your policymakers. Here’s a pre-made template that takes 30 seconds to send.
   

4. Donate. You could donate money to Manifund, where experts in AI safety choose how to best allocate your money.
   

5. Work. Use your career to help AI safety.

And if you’ve come this far, and you still think that AI existential risk isn’t worth paying attention to, I’d love to hear your feedback in the comments.

To humanity!
Mike