This spring, a U.S. special operations analyst had a question about a Chinese ship in the Middle East, so the analyst asked a chatbot. The chatbot mixed open-source reporting with classified signals intelligence and concluded the ship was carrying components for a nuclear weapons program. Then the analyst used AI again to write that conclusion up as a standard intelligence report, and the report went out across the military.
According to CNN’s exclusive, armed personnel were getting ready to board the ship and planes were already in the air before officials went back to the sourcing and found the problem. One source called the report “entirely false.” Another said it “almost started a war.”
Most of the coverage has focused on the chatbot getting the cargo wrong. I think the second AI step was worse. The first step produced a bad guess. The second step took that guess and formatted it like every finished intelligence product the chain of command reads each day, which is exactly the format people are trained to trust.
Ad tech already fought about this
When I sold data at Oracle Advertising, the smart buyers asked one question before they asked about price: how do you know? A segment called “in-market for an SUV” could mean someone who built a Tahoe on a dealer’s configurator last Tuesday. It could also mean someone whose zip code, age, and magazine subscriptions looked a lot like people who bought SUVs a few years ago. Same name in the taxonomy. Very different product. Only the first one describes something a person did.
The industry eventually made sellers write the difference down. The IAB Tech Lab’s Data Transparency Standard, the “nutrition label” for audience data, includes a field called Data Inclusion Methodology. The allowed answers are Observed/Known, Declared, Inferred, Derived, or Modeled. Other fields cover where the data came from, how often the segment refreshes, and whether it was padded out with lookalikes. Companies that want the compliance seal go through an annual audit to prove the labels are accurate.
That’s a lot of paperwork to protect a CPM. The worst outcome of a mislabeled segment is a car ad served to someone who takes the bus.
Nobody reading the report could tell what was a guess
Read the CNN account again with that label in mind. The chatbot started from reporting on the manifest that originated with U.S. Special Operations Command Pacific and fused it with other sources. On the IAB’s scale, its conclusion was Inferred at best. Then the formatting pass removed the one signal a reader needed. Once the output looked like every other report in the queue, nobody downstream had a reason to ask whether the nuclear claim came from something a person observed or from a model connecting dots.
The tool itself wasn’t going to save anyone. A former senior U.S. official told CNN that the military’s internal AI tools are “mostly just copies of the commercial stuff wearing lipstick.”
The pressure to move fast is real and on the record. The Pentagon’s AI strategy, released in January, calls for an “AI-first” force operating at “wartime speed,” and CNN reported the push is meant to put AI models “directly in the hands” of three million civilian and military personnel. Moving fast with AI is a defensible call. I’d make it too. What I’d change first is handing three million people a tool that writes finished-looking reports with no required field for how the conclusions were reached.
Credit where it’s due: somebody checked the sourcing before anyone set foot on that ship. That check is the whole ballgame, and it happened a lot later than anyone should be comfortable with.
Your team is doing the same thing with lower stakes
You’re not boarding ships. But look at what your team produced with AI this week. Call recaps land in the CRM as “the customer said,” and account research shows up in a QBR deck with a competitor’s pricing that came from who knows where. Each one starts as a model’s inference and ends as a clean document your VP forwards without a second look.
This is already normal behavior. KPMG and the University of Melbourne surveyed more than 48,000 people in 47 countries and found 66% of employees rely on AI output without evaluating its accuracy. And 57% say they hide their AI use and present AI-generated work as their own. That second number is the analyst’s formatting step, repeated across the workforce.
So steal the data label. Any AI-assisted document that’s going to drive a decision gets one line at the top that says which claims were observed (someone saw it, a system logged it, a customer said it on a recorded call) and which ones the AI inferred or modeled. It takes about ten seconds to write. It also gives the person reading it cover to ask “how do you know?” before they act, and that question is what stopped the boarding party.
If you sell AI tools, expect buyers who’ve been burned to start asking for provenance in the demo. Build the label into the product before procurement writes it into the RFP.
The ad industry built an audited label so nobody would waste a CPM on the wrong audience. A report that put warplanes in the air deserved at least the same paperwork.