The Human Was in the Loop. The Verification Was Not.
A chatbot-assisted intelligence report nearly sent US boarding teams onto a Chinese ship.
The next day, the president announced an “AI Force.” Those two events are easy to connect. But the interesting failure is not that the model got something wrong.
It is that a military intelligence process could no longer clearly distinguish between an assessment written by an analyst and one produced by a model and formatted to look like a finished intelligence product.
What happened — and what we actually know
On September 18, CNN reported that during the spring war with Iran, US forces came within hours of intercepting a Chinese-flagged vessel in the Middle East.
The intelligence assessment said the ship was carrying components linked to a nuclear weapons programme. Aircraft were already airborne. An armed boarding party was preparing. The operation was stopped when officials went back through the assessment and found that a chatbot had been used to produce it — and that the chatbot had misidentified the cargo. One source described the assessment as “entirely false.”
The next day, September 19, President Trump announced on Truth Social that he was creating an “AI Force” modelled on the Space Force, with an “AI Czar” to be named shortly. Reuters noted that the announcement gave no details about its structure, chain of command, size or mandate.
Those two stories were quickly paired in commentary as a contradiction: the military was institutionalising AI just after AI had nearly started a war.
That framing misses the more useful point. The incident happened months before the announcement. It was only disclosed the day before it.
The interesting connection is different. The military announced a new organisational structure without publicly answering the question the incident actually raises:
Who is responsible for checking what an AI system says, and what exactly are they supposed to check it against?
What is known, and what is not
Everything public about the incident comes from one CNN report based on four unnamed sources. The Pentagon and Special Operations Command Pacific did not comment. So the evidence is thin, and that matters when looking at the details.
According to the sources, an analyst at SOCPAC in Hawaii was working with intelligence on the ship’s manifest. The analyst asked a chatbot to combine open-source information about the vessel with classified signals intelligence and assess what the ship was carrying. The model concluded that the cargo included components related to nuclear weapons. The analyst then used the chatbot again, this time to turn that conclusion into a standard intelligence report. That report moved through military channels and was picked up by planners.
CNN could not establish what the cargo actually was, whether the chatbot was a commercial system or a government-provided tool, or exactly which verification step eventually caught the mistake. One source said there is no common standard across the US system for checking output from these tools. Another said this kind of hallucination was not unique.
Even from that limited record, two things are fairly clear. First, nobody was attacked. The system caught the mistake before the boarding took place. Second, the error traveled a long way before anyone caught it. And it traveled inside a document that looked exactly like the other intelligence products people were used to trusting.
Two queries — and why the second one matters more
The incident is usually described as a hallucination. In the narrow sense, that is true. The model produced a confident and specific false claim about the cargo. But language models producing confident false claims has been known since at least 2022. Anyone putting one into an intelligence workflow in 2026 already knows that risk exists. Treating the hallucination itself as the main failure is like blaming a sensor for having a known noise floor.
The more interesting failure happened in the second query. Intelligence products get much of their credibility from their form. A standard-format assessment, with the right classification markings, distributed through the right channel, and written by an analyst with the right access is trusted partly because the format implies that a process has already happened. Sources were checked. An analyst made a judgement. Someone reviewed it. The people receiving the product normally do not redo that work themselves. They read the form as evidence that someone else already did it. The chatbot changed that. The analyst asked the model to take its own conclusion and put it into the same format. The model could reproduce the appearance of a vetted intelligence product without reproducing the process that was supposed to stand behind it. The document that reached the planners looked checked. Nobody had actually checked the underlying conclusion. A human was involved at every stage. The human asked the questions and passed the result on. Nothing in the workflow made it clear that this was different from human analysis.
That is the real vulnerability. It is not simply that the model can be wrong. It is that the organization's trust signals were built around human work, and a fluent machine can pass through those signals without much difficulty. For years, provenance was a useful shortcut for judging quality. That shortcut stops working when nobody knows which sentences came from a model.
Confidence is not correctness
This is why the problem has to be built into procedure rather than left to individual judgement. A language model does not “believe” that a ship is carrying something. It generates text based on patterns in its training and the material it was given. When the evidence is ambiguous, contradictory or simply incomplete, the answer does not necessarily become more cautious. It can still produce a very confident assessment of what a plausible intelligence report would say.
The language stays fluent. The truth does not. And the model has no internal signal that tells the reader which situation they are looking at.
The usual analogy is a satnav confidently directing a driver into a river. The analogy works for one important reason: in both cases, the system can sound certain even when the underlying estimate is wrong. But the driver gets feedback almost immediately.
The intelligence consumer may not get it until after the operation. That difference is the core problem. It is also why simply telling people to “be careful with AI” is not enough.
What operators already know about bad sensors
Anyone who has flown FPV in an electronic warfare environment already deals with a version of this problem, usually without naming it. The video feed is a source. The telemetry is a source. The thermal image is a source. All of them can degrade in ways that do not look like degradation until something goes badly wrong.
A link-quality indicator can still look fine while the video is a second behind reality. A thermal hotspot can be a vehicle, a decoy or just a piece of warm metal in a field. A GNSS position can look smooth and precise and still be completely wrong.
The practical rule is not “never trust the instruments.” It is simpler: when two sources disagree, do not follow the one that looks more confident. Find out which one is wrong before you commit the airframe.
When there is no second source, you assume the first one can be wrong and act conservatively. The same principle applies when developing datalinks and thermal systems. The number the system gives you is not the first thing to trust.
It is the last thing you trust, after comparing it with something the system did not generate itself. This works in the cockpit because the consequences are immediate and personal, and because another source is often available within seconds.
Those conditions were missing in Hawaii. The analyst had one system generating both the analysis and the document. The cost of being wrong would have been paid by someone else. And the workflow did not require a comparison with an independent source.
The enemy gets a vote
There is another problem that has received much less attention. It is the adversarial one.
The analyst combined open-source information with signals intelligence. Open-source information is, by definition, information the adversary can influence.
During the same Iran war, AI-generated images of damage to the Fifth Fleet headquarters in Bahrain circulated within a day of the strike. A widely used chatbot was also found to be authenticating fabricated videos as genuine.
That creates a different risk. If chatbot-assisted fusion of OSINT and SIGINT becomes routine, an adversary no longer has to target the public. It can target the intelligence process itself. Seed open sources with plausible false information. Let the model combine that material with real classified data. The output can then look like a legitimate intelligence assessment while carrying the credibility of the real source and the falsehood from the poisoned one.
Nothing in the CNN report suggests that happened here. But the mechanism that produced an accidental false positive is also the mechanism an adversary could deliberately try to exploit.
The defence is the same in both cases: treat model output as untrusted input, no matter how polished the final document looks.
What an “AI Force” does not answer
Whatever the AI Force eventually becomes, the public record does not yet tell us much.
The proposal answers questions about organisation and acquisition: Who owns the programmes? Who sets the standards? Who reports to whom?
The Pentagon’s January 2026 AI Acceleration Strategy and the GenAI.mil platform, which the department says has been used by more than 1.3 million personnel, address a different question: How do you make these tools available, and how quickly can people use them? None of that answers the question raised by the SOCPAC incident.
That question is procedural and not especially exciting: When a model contributes to a product that could trigger an operation, who is required to check it? What do they check it against? Who has the authority to stop the product? And how is the model’s contribution recorded so the person checking the product knows what actually needs to be verified?
A new command can contain those rules. It does not create them automatically. And a structure built without them risks scaling the same failure instead of preventing it.
Doubting the machine has to be someone’s job
One line that circulated after the story was that the future is not AI replacing the analyst, but people trained to doubt AI. That is basically right. But as a slogan, it puts too much responsibility on the individual.
The whole incident shows why that is not enough. The analyst may have had doubts. The people receiving the report had no way to know that they should have been doubtful too.
The better answer is to make scepticism part of the process, with a clear owner. That means three things.
1. Mark AI-generated content at the sentence level
If a product contains model-generated analysis, the product should say so. It should also show which parts were generated by the model. That makes it possible for the reader to apply a different standard to those parts. The trade-off is obvious: the process becomes less smooth and less convenient.
2. Give someone independent authority to stop the product
There should be a person whose job is to verify the product before it can drive an operation. That person should not be the same person who generated the material. And they should be rewarded for catching errors, not for moving documents through the system quickly. That costs time. But speed is also the reason these tools are being adopted in the first place.
3. Keep a record of the interaction
The organisation should record what the model was asked and what it returned. That way, if the assessment is challenged later, people can reconstruct how it was produced instead of arguing about what happened. That costs storage. It creates classification and security problems. And it exposes the analyst to more scrutiny.
All three costs are real. Whether “doubting the machine” becomes a formal function depends on whether an organisation is willing to pay them.
Right now, many incentives point the other way. The strategy documents focus on acceleration. The platforms reward volume.
And a near miss that ends without a shot being fired is easy to describe as a success of the system rather than evidence that the system nearly failed.
What matters
The SOCPAC incident does not show that language models are too unreliable for intelligence work.
It shows something more uncomfortable: they may be reliable enough to be trusted by default.
A system that fails obviously gets checked. A system that produces fluent, well-formatted and mostly correct assessments will be checked less and less. Until the one time that matters.
For engineers building these systems, the goal should not simply be a model that hallucinates less. The output should be impossible to mistake for a vetted intelligence product — structurally, visibly and in its metadata.
For staff officers, the useful question is not only: “Is this right?”
It is: “What in this document was checked against something the model did not produce?” And for anyone watching the AI Force take shape, the more important thing to watch is not the org chart.
It is whether the organisation publishes a verification standard — and whether someone has the authority to say no. The operator's version of this rule is older than the technology. You do not commit on a single source. And you do not trust a source because it sounds certain.
The staff version has not been written yet. Until it is, the human in the loop is partly a formality. And the loop can close on its own.
