Another reason we're told not to worry is that these machines can't help but share our values because they will be literally extensions of ourselves.
They'll be grafted onto our brains, and we'll essentially become their limbic systems.
Now take a moment to consider that the safest and only prudent path forward, recommended, is to implant this technology directly into our brains.
Now, this may in fact be the safest and only prudent path forward,
but usually one's safety concerns about a technology have to be pretty much worked out before you stick it inside your head.
The deeper problem is that building superintelligent AI on its own seems likely to be easier than
building superintelligent AI and having the completed neuroscience that allows us to seamlessly integrate our minds with it.
And given that the companies and governments doing this work are likely to perceive themselves as being in a race against all others,
given that to win this race is to win the world, provided you don't destroy it in the next moment,
then it seems likely that whatever is easier to do will get done first.
Now, unfortunately, I don't have a solution to this problem, apart from recommending that more of us think about it.
I think we need something like a Manhattan Project on the topic of artificial intelligence.
Not to build it, because I think we'll inevitably do that,
but to understand how to avoid an arms race and to build it in a way that is aligned with our interests.
When you're talking about superintelligent AI that can make changes to itself,
it seems that we only have one chance to get the initial conditions right,
and even then we will need to absorb the economic and political consequences of getting them right.
But the moment we admit that information processing is the source of intelligence,
that some appropriate computational system is what the basis of intelligence is,
and we admit that we will improve these systems continuously,
and we admit that the horizon of cognition very likely far exceeds what we currently know,
then we have to admit that we are in the process of building some sort of god.
Now would be a good time to make sure it's a god we can live with. Thank you very much.