Among the many ethical considerations inherent in AGI is the issue of how to prevent such programs from causing harm. The debate among scholars is not just over how to prevent harm, but also how to define it.
The DeepMind survey authors emphasize that even a Level 5, artificial superintelligence might not actually be "autonomous." It might have cognitive capabilities but be constrained in its task execution for safety reasons.
The authors make the analogy to user interface design and human-computer interface design — it's up to creators of AGI, and society, to decide how much autonomy is given to such systems.
"Consider, for instance, the affordances of user interfaces for AGI systems," write Ringel Morris and team. "Increasing capabilities unlock new interaction paradigms, but do not determine them [emphasis the authors']. Rather, system designers and end-users will settle on a mode of human-AI interaction."
The DeepMind team emphasizes that whether there is harm or not is dependent not only on the capabilities unlocked in AGI but also on what kinds of interaction are designed into that AGI. A positive outcome, such as human superintelligence, can come with the right design of human-computer interaction.
Others see a broad and deep potential for harm for which there are no immediate answers.
Kurzweil writes: "Superintelligent AI entails a fundamentally different kind of peril — in fact, the primary peril. If AI is smarter than its human creators, it could potentially find a way around any precautionary measures that have been put in place. There is no general strategy that can definitively overcome that."
Similarly, Gary Marcus has argued that there is no plan to deal with any AGI that should arise. "If the fiasco that has been Gen AI has been any sign, self-regulation is a farce," wrote Marcus in a recent Substack, "and the US legislature has made almost no progress thus far in reining in Silicon Valley. It's imperative that we do better."