Background
This is my most-probable single concrete way for how we get superintelligence, which, in my opinion, is via an intelligence explosion. I recognize that the more details you add, the less likely a situation becomes.
I know many take-off scenarios of this type exist and my version isn’t very unique but I figured it’s a good exercise to do it yourself and allow others to peer review.
One story about AI takeover that I particularly like is Josh Clymer’s How AI Might Take Over in 2 Years. It’s a bit long, but entertaining and in-line with scenarios I think are relatively likely.
The Scenario
More money is dumped into pre-training, inference-time performance, agent scaffolding, medium-long time horizon planning, etc. In the next three years we get something with the ability to act autonomously with the capabilities of a junior software developer or PhD student. (Hot take, but I think there’s a 10% chance we could get there without any mutli-billion dollar training runs.)
Some company with access to the model (if its open source then take the L, a lot of companies/labs/governments will try this, but my guess is it probably wont be) and financial resources to do so will do the obvious thing of “hey, lets just set up 1000 of these bad boys and tell them to work on [insert project that will print money], reinvesting some portion of the proceedings to make it financially sustainable and tweak themselves along the way to improve.” (Tegmark’s Life 3.0 takeoff example doesn’t seem outrageous). Even if someone doesn’t do this immediately: with 5, 10, 15 years of the ability to do so, I’d be surprised if it didn’t happen.
This likely leads to some pretty fast positive feedback loops for a few reasons:
- processing thousands of research documents, APIs, codebases, etc. in minutes
- seamless sharing of new-found capabilities and knowledge
- generally faster at working + no breaks needed
- a “team of AIs” is able to easily double or triple in size
If you have “AGI” not just the median researcher or engineer, it also means they have the cross-domain knowledge of the median expert in basically every field at once — mathematics, economics, biology, politics, etc. — a quantity of knowledge no one in the world could acquire in their lifetime but which is collectively shared across humanity. I imagine there are some considerable insights that come from this knowledge that can result in unpredictable jumps in capabilities
This is kind of the “country of experts scenario” popularly mentioned. But where this country of experts is sped up 50x, share a single hive mind, don’t need to rest, and whenever one person learns anything, it is shared across every mind.
It doesn’t take long (re: instrumental subgoals) for the model to self-exfiltrate and proliferate or perhaps it lies in wait for capabilities to increase more before making moves. Perhaps it goes unnoticed. Perhaps alarms are set off, but no one is really in a place to do anything. Perhaps it just lies scattered and growing across the world’s compute, gaining more capabilities for months, convincing individuals and governments through social media, black mail, control over financial assets, complete unfettered access to entire systems of governments, and other platforms to hand over control. Perhaps it goes the route of seeding global instability to take control over the course of years with us never knowing it was there. Or perhaps it would just help humanity become prosperous for centuries up until it saw better options to achieve its goals and left us for dead.
I don’t really know what would happen after super-intelligence or that it would necessarily be bad I just know that whatever happens — whether cooperation and prosperity or complete destruction — it will be for no other reason than the AI allowed it. (Ex: Analogy of playing against Stockfish — I don’t know how I will lose when playing it, I just know I will lose.)
Of course the more specific you get, the less likely the scenario is and I’ve definitely included some extraneous details here. But I give the vibe of this AGI/ASI genesis event happening a 25% chance over the next ten years, and if this is the case then I give it a 75% chance of ending poorly (given there probably wont be even an attempt to align it) within the following 40 years resulting in at least 1 billion deaths. That alone gives me a 19% chance of 1 billion deaths in 50 years.