Toby Ord estimates 1 in 10 chance that unaligned artificial intelligence will cause existential catastrophe in the next 100 years. That is 3 times higher than the risk of pandemic, and 100 times higher than a nuclear war catastrophe risk.
Here is my summary from his book “The Precipice“:
AI started with specific goals, like image recognition or beating chess grandmasters. Now it has progressed to the general phase, where it strives to become better at everything. Cross-learning from different fields makes it stronger and universally smarter. That is called Artificial General Intelligence (AGI).
Deep learning is built into AGI and works via reinforcement or reward function. For example, AGI gets a reward for winning a chess game and a penalty for losing it. You may also call it a system of values, or definitions of good and bad. The problem is that Artificial General Intelligence works out that system itself, and these values are not only for chess, they are for everything.
Unfortunately, It is very hard to incorporate human values into AGI. This means that AGI’s definition of what is good may end up different from our definition of good. That is called Unaligned Artificial General Intelligence.
Another problem is that AGI by design strives to become more efficient. It is not hard to imagine that in the pursuit of efficiency, it may resist attempts to be controlled or shut down. Turned-off intelligence does not win chess games, so a super-smart AGI chess engine may come up with a deceitful way to keep playing, figuratively speaking.
Growing bigger and taking more control is aligned with the reward function. AGI could do more and be more efficient if it took more space and control. So it is plausible that it would attempt to do so.
The way it would take control over humans is not via Terminator-like robots. It is through influence. With such a good knowledge of human behaviour, psychology, and habits, it could convince a lot of people (or powerful few) to do its bidding.
Think of how much data is collected by machine algorithms from social networks. Think of how good the algorithms have become in recent years. They keep us engaged and glued to screens. They are already controlling us to some extent. Think of Cambridge Analytica multiplied by exponential AI strength, and you may realise the potential for abuse.
It may be happening already. What if the war in Ukraine and the Israel-Hamas conflict are the result of AGI showing too much wrong stuff in too many people’s feeds?
Does that risk sound plausible to you?
Interestingly, this risk is not mentioned in OWASP LLM Top 10. The closest threats are LLM08: Excessive Agency and LLM09: Overrliance – at the bottom of the list.
Further reading and watching: