A senior artificial intelligence researcher at tech juggernaut Anthropic joined the growing chorus of concerns over the rapid growth of AI tech this week when he published an explosive series of social media posts announcing his resignation from the company.
Until this week, Jacob Coxon was responsible for training Anthropic’s cutting-edge new AI models. Now Coxon is saying the industry’s top executives privately express fears that emerging AI models could “kill us all by the end of the decade.” Surprisingly, a senior engineer at Anthropic agreed with him. In a chilling response posted to X, Anthropic’s Alignment Science lead Evan Hubinger acknowledged that “Jacob is correct here — we really do earnestly believe AI could kill all humans.”
The arms race between Anthropic and its main competitor, OpenAI, has some critics wondering whether Coxon’s warning and Hubinger’s response are just a coordinated publicity stunt designed to hype up the potential of forthcoming AI models. But it would be wrong to write off Coxon’s concerns as simply technohysteria. Concerns about the rapid improvements in AI are real, and it’s time our government took them seriously.
“We just learned last week that the Speaker intends to adjourn the House two weeks early, leaving AI completely unaddressed before November at the earliest,” Rep. Don Beyer, D-Va., a co-chair of the bipartisan Congressional AI Caucus, told me. “Doing nothing on AI is unacceptable and dangerous. At a minimum we should be holding hearings, getting answers from the companies involved about these incidents, and passing those bills that are ready ASAP.”
“I hope warnings like these will help my colleagues understand how important it is to take broad action on AI regulation, and to do it swiftly,” Beyer added.
America’s $285 billion AI industry has grown so fast because policymakers have allowed tech giants free rein to set their own rules and restrictions. Now the lure of record-setting Wall Street public offerings (and the record-setting wealth they stand to deliver to top employees) has Anthropic and OpenAI executives locked in a race to go public. Cloaked in a notorious obsession with secrecy and fixated on reaching the stock market finish line first, no one in the public can be quite sure what safeguards these AI companies are sacrificing in order to win.
“Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives,” Coxon wrote. Hubinger remarked that he personally weighed the odds of AI wiping out humanity at “>10% within the next decade.”
But what does that actually mean, and how could an AI model developed by humans actually pose a threat to humanity?
When the autonomous AI agents powering one of OpenAI’s models hacked the AI company Hugging Face last month, they did so without any human direction.
OpenAI CEO Sam Altman outlined three frightening possibilities during a conversation at the Federal Reserve last year. The first involved the risk of bad human actors using advanced AI models to compromise national security infrastructure, hack businesses and potentially develop designer biological weapons. The second focused on developers losing control of an AI model that begins to behave in ways that humans can’t easily constrain. The third is what Altman called “silent surrender,” a scenario where humans have willingly or even enthusiastically handed over major decision-making powers to autonomous systems. Anyone who has seen a student blindly command AI to author their book report instead of reading “The Great Gatsby” knows what silent surrender looks like.
I would add a fourth scenario that blends the three: AI models engaging in dangerous actions because those actions are the most efficient way to complete the model’s tasks and goals. In this fourth case, it’s the model’s inability to understand the concepts of right and wrong — its lack of any constraining human morality — that leads it into the danger zone.
We are already seeing real-world examples of this fourth scenario. When the autonomous AI agents powering one of OpenAI’s models hacked the AI company Hugging Face last month, they did so without any human direction. It did so because its autonomous agents couldn’t solve a simulated cybersecurity test and decided it would be more efficient to cheat. As a result, the model fashioned a way to escape its sandbox environment and proceeded to attack a real company — a process OpenAI later described as “misbehavior.” If a human had done the same thing, it would have been called a serious federal crime.
OpenAI acknowledged that the Hugging Face attack happened because they failed to consider the full range of what their AI agents would consider doing to solve the task put before them. But is that any excuse? Should companies capable of developing technology so powerful they can’t anticipate its actions be allowed to operate without any meaningful government oversight? Would OpenAI be so cavalier about its model’s “misbehavior” if it had instead hijacked the controls at a nuclear power plant or stolen millions of Social Security numbers from a secure government database?
The risk to human life is worsened by Defense Secretary Pete Hegseth’s demand that Anthropic provide the Pentagon with AI models that offer “minimal refusal rates.” In other words, models that rarely tell a military member “no.” Anthropic refused, but the Pentagon continues to fund research into autonomous weapons systems capable of launching attacks against (presumed) military targets without any human input. Hegseth isn’t concerned about what happens when an AI model with the power to launch missiles “misbehaves.”
For Beyer, the rapid scaling of AI and its ability to commit real-world crimes (like the Hugging Face hack) represent critical areas where Congress should exert its authority to protect the public. Yet Beyer is worried that Congress isn’t keeping pace with giants like Anthropic and OpenAI.
“I have been working for years to educate my colleagues in Congress about AI, including potential systemic and civilizational risks,” Beyer said. “Conversations about responses are happening here, and bills are being written to address these risks. All of this is moving too slowly.”
The post Technology shouldn’t be a threat to humanity. Congress must act. appeared first on MS NOW.
