#25: Nobody's Thinking Enough About AI

A very strange thing is happening. Technology is getting wildly better, faster than ever before. And while people are excited about the new AI products in the headlines, almost nobody is willing to look a few years into the future and ask seriously what this means. The following things happened in just the last few weeks:

• President Trump announced Project Stargate, a $500B investment in infrastructure for OpenAI.

• Dario Amodei, founder of Anthropic, claimed¹ we are two to three years out from Artificial Superintelligence.

• OpenAI released Deep Research, which is capable of creating in-depth, graduate-level research reports — thousands of words — in minutes.

• OpenAI released Operator Mode, a computer use model that can use your browser to execute tasks just as a person would.

• DeepSeek released an open-source model that is roughly on par with OpenAI’s O1, and claimed (misleadingly[2][3]) that it cost only $6M to train.

Many well-respected AI industry leaders are now expressing fairly short timelines: on the order of artificial general intelligence (AGI) within 5 years.² These aren’t outlier views, either. On the technologist prediction market Metaculus, 1484 people have cast their bets, and the median one is for May of 2030. The upper bound has been steadily coming down. “Later than 2038” is coming to be a fringe view. And of course, some think there’s a good chance we’ll get there next year.

Whether it’s five, ten, or fifteen years — take that seriously, and let it sink in for a moment. That’s soon. We might be just a few years out from sharing the planet with software that is infinitely replicable and can do the exact same mental tasks as you and I. The implications are immense. For a start, it’d be the biggest job turnover cycle in history. Even if the chance of this happening were small — not zero — that still seems worth thinking seriously about, in the same way that we prepare for other contingency events. But despite industry leaders being very clear about where things stand,³ there’s basically no public discussion about this. Not a peep from policymakers;⁴ the Overton window isn’t there yet. This seems like a real miss.

This is a long article, so I’ll put the take-aways explicitly up front:

1. Progress in AI is much faster than you think.

2. How to deal with AI-driven social and economic change is probably the only public policy question that matters right now.

Speed

The first thing to recognize in AI progress is that it has already happened much faster than people expected. I remember very well the 2015-era hype cycle for machine learning:⁵ breakthroughs using neural networks and generative techniques were finally returning rapid progress to a field that had stalled for years. But researchers still quoted AGI timelines for the year 2050 or 2100, with many shying away from saying that it’s possible at all.

At that time, the idea of AGI by 2030 would’ve been received as a little kooky. Now it is mainstream. Only famously-out-there-futurist Ray Kurzweil seems to have gotten it right: he bet that the Turing Test would be passed by 2030. There’s a prediction market on this: in early 2020, bettors were pricing this at 20%. Now it’s at 80%.⁶

During this time, we’ve kept coming up with AI tests and benchmarks, blasting through them — attaining parity with human experts — and then coming up with new ones. You know you’re making headway toward AGI when it’s getting harder and harder to come up with tests that distinguish human from AI performance.

There’s now the cutely-named Humanity’s Last Exam, a collection of 3,000 difficult questions across a hundred fields. While I think the exponential plot of its performance improvements is misleading,⁷ the progress is undeniable:

What’s going on here is that progress toward AI follows an exponential growth curve.⁸ Unfortunately, even highly intelligent, technical people have no intuition at all for exponential growth. Most people’s imaginations are stuck in linear growth — and so, time and time again, people underestimate how quickly growth can compound.⁹

Cartoon Source

Why Aren’t People Talking About It?

Though the headlines are everywhere, to most people they are still abstract. If they’re not using AI products, then they won’t develop intuition for those coming changes. Moreover, those changes may still feel far away because technology has historically been slow to percolate into the real world:¹⁰ plenty of things are still done manually even though software exists to do it. Even the internet took years to go mainstream, and many more years to start truly reconfiguring the world around itself.¹¹

But these may not be the right analogies. Change in the physical world comes slowly, where logistical constraints exist. But change in the digital world — where we all now spend key parts of our lives — can come very, very, quickly.

People already underestimate the speed and scope of present change because “AI” has been such a buzzword for over two years now.¹² People aren’t noticing how much better these technologies have gotten during that time, because the form factor of interacting with them (i.e. for most people, the ChatGPT window) hasn’t changed. It still feels the same to the consumer,¹³ and the cycle of consumers being amazed by new technology to them totally taking it for granted is very short.

Finally, perhaps the very idea of AGI is still too “out there” for people to discuss. In a similar way, in most of the world the idea of a self-driving car is still space-age science fiction. But in San Francisco, people are taking self-driving cars every single day. The technology which always seemed forever away is finally arriving.

It Gets Personal

All these changes seem abstract and far-away until they knock on your door. It’s really hard to appreciate or develop intuition for this progress until you see software doing the tasks that you take pride in: then, suddenly, it’s crystal-clear. In the next few years, most knowledge workers will have an uncanny realization when they’re looking at a screen and saying “wait, this thing can do what I do.”

Everyone has their Lee Sedol moment.¹⁴

I had mine recently.

Deep Research

I’ve spent a lot of time in my life doing research. To that end, OpenAI’s Deep Research is amazing. It’s basically a longer-running version of ChatGPT — instead of visiting one or two websites and returning a short answer in thirty seconds, it visits dozens of websites and returns an essay-length answer in ten minutes. Objectively, it does a great job. (Example research projects in the footnotes.¹⁵)

Deep Research hugely decreases the intellectual “activation energy” required to learn about something new: instead of having to muster the effort and spend many hours combing through resources to synthesize details, OpenAI can do it all for me. This is greatly empowering¹⁶ — but of course, at the same time, clearly the clock is now ticking for lots of highly paid knowledge work. If we’ve gone from hallucinating-all-the-time early ChatGPT to this in just two years, where will be in another two?¹⁷

Cursor AI

I build software. When I was nineteen years old, I decided to make that my primary intellectual and professional quest. Twelve years later, I’ve become quite good at it. I’ve made things that I am proud of, and in some cases have been used by millions of people. Yet my experiences using AI to write code have put the writing on the wall: I have probably already written the majority of code that I will write in my lifetime. The skill that I have spent many years cultivating will slowly but surely become antiquated.

I had previously tried writing software using ChatGPT or Claude as an assistant to generate and modify code for me. But using Cursor with Claude 3.5-Sonnet was a much better, more fluid experience. Like Deep Research, it dramatically decreased the “activation energy” for starting a new project, and it easily generated hundreds of lines of complex-but-shallow code.¹⁸ These are the parts of the work that would be most time-consuming to me, and it handled them with relative ease. This is even though Cursor is still early as a product: there are lots of bugs, it makes plenty of mistakes, and doesn’t even leverage standard programming tools like linting or typing by default. There’s a lot of room for it to get a lot better very quickly.

Funnily enough, many programmers don’t get this. In what seems to be a head-in-the-sand-defense, many are looking at Cursor as it is today and saying “well, that’s not as good as me!” unwilling to imagine just another three years of improvements. I’m hearing things like “well, you have to learn how to prompt it…” and “it can’t handle deep, complex problems” and “it can never understand a codebase as deeply as me…”

But this is obvious and willful cope from people who should know better: of course these are solvable problems. Each of these objections will fall as AI-assisted code editors integrate better tools and deploy more powerful, longer-context, models that leverage more test-time compute. This seems inevitable to me. Once more, even smart, technical people do not necessarily have the intuition for exponential growth.

Beware Dismissing the Questions

Until 2022, a common attitude was that AI could never do creative work — that creativity was the domain of humans; that it required some special element¹⁹ that a machine could not replicate. This was popular to say and comforting to think, but now we know it also wasn’t true.²⁰

But maybe we always knew it wasn’t true: if you had really pressed this question ten or twenty years ago, you wouldn’t have gotten a good answer. People were dismissing these AI questions with lazy, hand-waving answers. The reality they were trying to avoid was very simple: if a human activity really boils down to thinking and AI can think, then that activity can be done by AI. Understandably, people want to believe that what they do is special and not possible to automate. They won’t believe otherwise until they see a machine do it. We will see this play out many times over the coming decades: people wishfully asserting that something is unique to humans, and then eventually finding that may not be so.

As we think seriously about the impacts of AI, there are plenty of these dismissive answers that are worth pressing. For example, people say not to be concerned about AI taking over jobs, because we’ve historically always created more, higher-leverage jobs as we’ve innovated old ones away. But there’s just no reason for that to hold ad infinitum. Why couldn’t AI perform those higher-leverage jobs, too?

What Should We Be Thinking About?

In the long run, AGI will transform our world, just like the industrial revolution did long ago. I think this can go very well for us, and the right attitude is cautious optimism. But no transformation of this size is without friction. While it’s not worthwhile to try to predict second- or third-order consequences (there are just too many variables at play), there are a few things that are straightforwardly predictable:

1. Many jobs will cease to exist, and be completed by AI instead.

   1. Even if some of these jobs are replaced by new jobs, the cycle of job replacement still means transient unemployment en masse.

2. People will generally have more and more free time.²¹

3. On the internet, AI activity will vastly outnumber human activity.

   1. If we believe that the distinction between human and machine matters, then working on proof-of-humanity seems important.

I’m not an AI Safety guy. I think AI will be fine. But I think that there is some danger of conflict between humans if we mess up the transition. For example, it’s easy to imagine a case where a rapid loss of jobs turns into social unrest, which spirals into something really ugly. It wouldn’t be the first time that people have felt threatened by automation, or that economic and social volatility is exploited by demagogues.

What Should Policymakers Think About?

First, please note that public policy moves on slow timelines. Change takes years to enact. If we’re taking seriously the idea of AGI by 2030, to a policymaker that’s basically tomorrow. We need to get started. To me, there are several tasks ahead:

1. The impacts of AI are mostly absent from contemporary policy discussions. This needs to change as quickly as possible.

   1. For a start, most policy issues will be affected in some way by AI. Simply asking “how will AI affect this topic?” will be a good way to start introducing the discussion at all levels, while also preparing for upcoming changes. If you’re debating policy X, it’s always worth reframing: what does X look like in a world where we have intelligence too cheap to meter?

2. Secondly, we need to begin discussing the impacts of AGI by themselves, not just as a framing for contemporary issues.

   1. We need to solve for the economic challenges of the transition to AGI. This mostly means preparing for rapid turnover within the labor market, and (in my opinion) for a rapidly shrinking labor force participation rate.

   2. We need to solve for the social challenges of the the transition to AGI. On the one hand, there will be some adversarial behavior to remedy. (Think AI-enabled spam and scams.) On the other hand, people will need to feel security, significance, and connection in a world where their labor or intelligence is suddenly no longer so special.

3. We need to internalize that these questions are way more important than almost any other contemporary public policy debate. This is where we need to spend our time and effort. Topics like immigration, climate change, income taxes, or whatever else, do matter, but not nearly as much as the big change coming up. And all these topics will look different in a world with AGI, anyway.

4. Finally, we need to think deeply and carefully about what’s going to happen. Given the difficulty of predicting the future: if in doubt, under-regulate rather than over-regulate.

I don’t have all the answers. My purpose with this essay is to get people thinking about these questions and appreciating their urgency. While there are some clear first-order action items, like needing to provide additional dignified ways for people to off-ramp from economic production,²² I remain suspect of easy answers that dismiss the questions: you can’t just say “UBI” and call it a day. You might be right in the long run, but it would gloss over the very real near-term challenges of getting there.

These are big topics — almost overwhelming in scope. Getting started won’t be easy. And stated in the abstract, they might not feel so urgent. But we live in a world that is built around the unique value of human intelligence and labor everywhere you look: from the thirty-year mortgage to people’s identities being wrapped up in their vocations, there are meaningful changes incoming for fundamental aspects of our society. They are manageable changes, of course, but they deserve careful preparation. I can’t think of anything more important than getting the AI transition right.

---

Appendix: Clearly We Have Not Hit The Wall

A few months ago, there was a moment of doubt about whether we had hit “the wall” in improving AI models: some people speculated that the scaling laws might not continue to hold, and that we might be in for an era of diminishing returns.

Right now, this doubt seems to be refuted. Progress is as fast as ever, though big gains seem to be coming from areas outside pure scaling. It’s worth revisiting Leopold Aschenbrenner’s Situational Awareness: while most of his text focuses on scaling laws, he points out that several orders of magnitude could be gained just from “unhobbling” the models by algorithmic improvements, and moving more resources to test-time compute. He cautiously suggests that it might be possible to reach AGI just via these two. My hunch is that this is about right. Andrej Karpathy once said that we might observe intelligence in a <1B parameter model, and while it’s still early, practical discoveries are pointing this way, too.

Some people are (maybe wishfully) assuming that the big burst of progress is done and will level off from here. I don’t think that’s true. At least right now, it looks like the exponential growth is uninterrupted, and there is no reason to expect it to slow down.

1. There’s a temptation to say that he’s talking his book, of course. But unlike how I view some other major AI entrepreneurs, I take Amodei quite literally. I think he takes a pretty straightforward academic view and I haven’t noticed him oversell in any other areas.

2. For a couple of predictions:

• In late 2024, Gwern gave it another 2-3 years.

• In early 2023, John Carmack quoted 50% by 2030. I assume he’s revised down since.

• Elon Musk has quoted 2029 on several occasions.

• Sam Altman has suggested we could have AGI in the 2020s on several occasions.

• Shane Legg quoted 50% by 2028.

• Demis Hassabis has said we’re “on track” for AGI by 2030.

• Vinod Khosla suggested a timeline of 2030.

• Jensen Huang suggested we’d have it by 2029, depending on definitions.

3. Not to mention the considerable lobbying efforts by Sam Altman and others! It’s remarkable to me that he’s met with basically every EU policy leader, not to mention their equivalents in the US, but none of them have come forward with any discussion about how to manage economic and societal change that is to follow.

4. There’s plenty of discussion about AI as a national security interest in the US, and from a misguided regulatory perspective in the EU. But the domestic economic/social policy aspect is totally missing.

5. I remember it vividly because I was there, and it was very important to me. I started taking AI/ML classes in the Spring of 2013, and that became the core focus of my undergraduate studies.

6. In my opinion, we’ve pretty well already passed the Turing Test, and the “80%” mostly reflects the precise details of the criteria for resolution of Kurzweil’s bet.

7. OpenAI Deep Research has access to in-depth web search, and some of the earlier models don’t. If they had a search integration, maybe their performance would be better and the trend would look less exponential. I think the right way to view this is either as a step-function-change with Deep Research, or as a less steep exponential if you provided web search to the earlier models, too.

8. I can think of two factors at play: first, to the extent that progress toward AI is compute-limited, the total amount of available compute seems to be increasing exponentially. (You can think of this as a kind of variant on Moore’s law.) Second, progress in AI is helpful toward creating more progress in AI. This self-reinforcing dynamic produces exponential growth by definition.

9. There are many observations on this topic that fit well. I like one from Bill Gates: We always overestimate the change that will occur in the next two years and underestimate the change that will occur in the next ten.

10. “75-95% of the productivity benefits of new technologies not from initial commercialization, but rather realized over years [decades] of diffuse implementation and incremental improvements / adaptations.” from Jonathon Sine quoting James Bessen’s Learning by Doing

11. The web opened to the public in 1991, and truly began gaining consumer adoption in 1993 as websites became commonplace. While there was initial excitement, true penetration took a long time: even by 2005 only 66% of Americans had internet access. (Global use lagged even further behind; Americans were early adopters.) I remember the 00’s as years where the online world was still considered something of a novelty; there was a kind of the-internet-is-not-real-life attitude. Ten years later, the opposite was true: everything’s online, and in many cases, doing things in the physical world was antiquated.

12. ChatGPT was released on November 30, 2022.

13. Note that the current ChatGPT is way better than when it was first released! It’s come a long way! But consumers struggle to notice this because it’s getting better in repeated incremental changes over time. If you hooked up a chat console to the now-deprecated GPT-3 API, you’d be shocked by how immature it was compared to what we have today.

14. What this specifically refers to is Lee Sedol, one of the strongest Go players in the world, losing to Google’s AlphaGo in 2016. When he recognized that the AI was vastly stronger than him — and could never be beaten as it would only improve further — he retired from the sport.

15. Below are some examples for you: a variety of research projects I had it run on topics I was curious about.

• US Dollar Positioning Under a Potential Trump Administration

• ServiceNow: Business Model, Products, Market Position, and Technology

• Email Provider Pricing Comparison (Transactional Email Sending)

• Ringo Starr’s Drumming – Expert Evaluations and Legacy

• Cultural Formation of Germany: Prussia and the Smaller States

• Prussia and the Baltic German Communities (18th–19th Centuries)

• Wittgenstein’s Philosophy of Mathematics and Finitism

16. At the margin, this means I am now learning about things that I would never have the time to otherwise. When I was younger, I always dreamed of hiring a full-time research assistant to dig into all my curiosities. It looks like this will no longer be necessary.

17. This is not a rhetorical question! Seriously, think about it and try to come up with an answer.

18. By “complex-but-shallow” I mean logic that is complicated to write, but doesn’t make a lot of nested calls or carry side-effects that need to be handled in code elsewhere. Frontend applications are full of these things: components that require particular styling and event handlers are great examples.

19. A soul, if you are so inclined!

20. You may have already seen this in generations from Midjourney or Pika or ChatGPT. Even if you don’t like their output, it is undeniably creative. But you can find this creativity in other fields, too. For example, take the AlphaGeometry2 paper: “our geometry experts and IMO medalists consider many AlphaGeometry solutions to exhibit superhuman creativity.”

21. Some argue that historically, jobs have always gotten replaced by higher-leverage ones. But this is deceptive. The amount of leisure time that’s available to people has been steadily rising, and the percentage of the population that performs economically valuable work has been slowly decreasing for decades.

22. Many of these off-ramps already exist in subtle or unofficial ways. In the US, twelve million working-age adults receive some form of federal disability benefits, and do not work at all, or only work part-time. In the EU, the anecdotal scheme for young people who are having difficulty finding employment seems to be a prolonged stay in higher education — second Bachelors’ or Masters’ degrees are becoming commonplace.