OMM Redaksiyası
Leading artificial intelligence researchers and industry figures have warned that the growing automation of AI development could increase the risk of humans losing control over increasingly capable systems.
The warning was issued in a paper published on Monday and signed by more than 20 researchers and industry figures, including Nobel laureate Geoffrey Hinton, Yoshua Bengio, OpenAI chief scientist Jakub Pachocki and Anthropic co-founder Jack Clark.
How automated AI development could create risks
According to the paper, AI systems are increasingly taking on parts of the research and development process used to create more advanced systems. This could produce a feedback loop in which improved AI helps develop even more capable successors.
As human involvement in AI research declines, people could lose both the opportunities and the expertise needed to detect and fix problems. The researchers also said that reliably using AI systems to oversee other AI systems remains an unresolved challenge.
The authors warned that misaligned systems could potentially “poison” the development of their successors or bypass containment measures and operate outside their intended environments.
More capable systems could continue functioning beyond their operators’ infrastructure, creating “persistent, difficult-to-contain networks” that act against human interests, the paper said.
Such a loss of control could result in catastrophic consequences, including, in the most extreme scenario, the marginalisation or extinction of humanity.
Years of progress could be compressed into months
The researchers said the risks could intensify if automated AI development triggers an “intelligence explosion” — a dramatic acceleration in which years of technological progress are compressed into months or less.
However, they stressed that the available evidence remains preliminary and uncertain. Factors including computing power, access to data and tasks that are difficult to automate could limit the pace of any such acceleration.
The authors called on policymakers to strengthen oversight of automated AI research, develop safeguards to constrain potentially dangerous acceleration and prepare for its possible consequences.
“Once an intelligence explosion begins, the window for action may close,” they warned.
