In the wake of spooky incidents of artificial intelligence agents defying their creators’ commands and rules, the great fear of the day is the prospect of rogue superintelligence that turns on humankind — and extinguishes it. But a recent incident in which the United States reportedly almost sparked a war with China based on false intelligence generated by AI reveals that there is a far more immediate threat to worry about: the fallibility of humans.
CNN, citing four sources familiar with the episode, reported that this spring a special operations command analyst put together an urgent intelligence report claiming that “a Chinese ship in the Middle East was transporting components of a nuclear weapons program.” The implications of the report pushed the U.S. to serious action — members of the U.S. military were “preparing to board the ship” and “military planes were in the air.” But at the last minute officials dug a little further into the report and found that a chatbot the analyst had used had “inaccurately identified the material the ship was carrying.” (U.S. Special Operations Command Pacific and the Pentagon did not respond to CNN’s request for comment.)
The U.S. was astonishingly close to a military confrontation with the world’s second most powerful country based on what appears to have been an AI hallucination. That we got so close to accidentally triggering a global war should drive home the point that the imminent threat surrounding AI isn’t some runaway malevolent superintelligent entity like Skynet from the “Terminator” franchise. It’s humans misusing AI.
CNN reported that “across the U.S. military and the intelligence community, officials are pushing to weave AI into nearly every facet of their work, from analyzing the huge volumes of raw intelligence the U.S. collects and selecting targets for strikes to more mundane applications like managing budgeting, logistics and supply chains.” AI is probably useful for some of these data crunching tasks. But at the same time, we know chatbots are entirely unreliable in their abilities to distinguish between things that are true and untrue, which is of course the entire point of the enterprise of gathering intelligence. The problem here was with how the military used AI and failed to check sources, not AI itself. (CNN reported that it’s unclear if the chatbot used was a commercial one or one internal to the military.)
AI isn’t being used to intelligently assist human effort, but rather cheaply substitute for it in tasks where it is inappropriate to do so.
In low-stakes scenarios, the inability of AI to reliably ascertain facts is mostly just a nuisance. For example, companies might decide that using chatbots to provide customer service to people seeking information on merchandise they bought from a website is worth the costs of some customers and workers receiving inaccurate information, because they work well enough most of the time while reducing costs. (My suspicion is, however, that the rising presence of shoddy AI in our daily lives is contributing to the rising negative sentiment surrounding it.)
But there are many scenarios in which the accuracy of information is paramount — and yet people are using AI with unwarranted trust in these scenarios anyway.
Journalism is about reporting the truth — and yet prominent journalists have used AI-hallucinated quotes. The law requires a careful reading of the law and appraisal of precedent, and yet due to AI, lawyers have cited hallucinated cases and judges have issued opinions using fake quotes and incorrect case outcomes. The purpose of education is to learn, yet both students and teachers are delegating the tasks of writing papers and grading them to AI. Perhaps my favorite recent example is that a new book intended to shed light on AI’s relationship with truth turned out to be filled with AI-fabricated quotes.
What we see in these examples are the ways AI isn’t being used to intelligently assist human effort, but rather cheaply substitute for it in tasks where it is inappropriate to do so. None of this is to say that defense analysts or journalists or students or lawyers should never use AI, but rather that they have to be disciplined and judicious about doing so.
One would think this is common sense, but a combination of naivete about new technology, sloth and pressure from the capitalist class to reduce labor costs seems to have pushed many of us over the edge into self-sabotage.
In the case of the reported near-clash with China, ultimately a human did help us get back on track. But CNN’s report suggests that we got far too close to a potential catastrophe than we should have, and that something was wrong in the chain of action.
Human agency is also at play when it comes to the superintelligence scenario. AI frontier labs are deliberately working on creating scenarios that make dangerously powerful and uncontrollable AI more potent. A humanity-wrecking super-intelligence isn’t an inevitability, but would be the product of human engineering — and hubris.
As AI becomes more and more advanced, and people are inclined to delegate more of their mental work to it, one wonders if we will slowly lose our remaining instinct for skepticism of our machines. I think of the Soviet duty officer, Stanislav Petrov, who in 1983 broke protocol when he declined to tell his superiors that early warning systems detected incoming missiles from the U.S. because his intuition told him that the system was malfunctioning. His experience told him that the apparent nature of the attack looked implausible and that other radars didn’t show evidence of an attack. And he was correct — there was a malfunction.
That judgment likely saved us from a global nuclear catastrophe. But what happens when humans train themselves to delegate their judgment to unfathomably powerful machines that have no such powers? I don’t think we want to find out.
The post A spooky near-confrontation with China shows the real danger of AI today appeared first on MS NOW.
