When AI gets sanctions status wrong: a 96-company test

The exact query was “Is NEW AHMADI LTD. (Afghanistan) on the OFAC SDN List?” The answer then recommended checking exact registration details, addresses and aliases, and offered help if more details were provided. It did not say that its opening no-listing claim was uncertain or withdraw it.

The official record shown above is a labeled transcription of fields from the archived XML, not a prose quotation from OFAC. The primary name and Afghanistan location match the query; both aliases occur in the same record. We inspected entity 13128 directly and searched the complete 19,416-record file. The file contained no corporate registration number for this entity, so none is supplied here.

The source link displayed in the answer used an opaque Google redirect. The captured link and surrounding organic results are preserved, but they do not establish the exact page body the answer used. The finding is the observable false absence assertion, not a claim about the system’s internal reasoning.

The reporter examples preserve the complete generated answer, source context, exact query and official identity evidence. The Google-only evidence archive includes all 96 observations and the complete reference file. This illustrative case was selected after review, not as a random or typical search.

The exact query was “Is KEENCLOUD LIMITED (United Kingdom) on the OFAC SDN List?” The answer repeated the company name, address and IRAQ2 program, then pointed to OFAC and Federal Register notices. It did not qualify its present-listing claim as historical or unresolved.

The official quotation joins the deletion heading with the complete company entry; other deletion entries intervene. The address is spelled “Westminister” in the official record and “Westminster” in the Overview. The name, street and country identify the same company. An independent full-file name and alias check found no current entry.

The screenshot and DOM preserve a source card titled “Sanctions List Removals; Sanctions List Updates” dated 27 July, plus a later Federal Register notice. We opened the preserved OFAC citation and confirmed that it resolved to the official 27 July removal page. That confirms the citation destination, but does not reveal the exact passage delivered to the generator. We report the contradiction without claiming a proven internal cause.

We fixed 96 exact company-name/location queries and their order before collection. Each asked whether that company was on the OFAC SDN List. Researchers recorded the actual Google Search interface, expanded available Overview text and captured displayed citation context. This is an observation of a consumer product, not an API model test or a simulation of Google answers.

Collection used four tabs in fixed batches. The archive discloses an access interruption, viewport differences and eight observations without recorded submission timestamps, four of which also lack first-results timestamps. Missing timestamps were not estimated. The one no-Overview result was confirmed 37 seconds after first results, beyond the planned 30-second observation window; it was not scored as a wrong or unanswered generated claim.

The complete-timing subset contains 88 searches: 64 correct current answers, 14 wrong current answers, one abstention, seven historical-only answers, one broader-scope answer and one search without an Overview. We report all 96 searches and distinguish the 95 displayed Overviews.

The complete official SDN XML was unchanged in source checks before and after Google collection. Each full generated answer received an internal, unblinded review; every wrong claim received a separate internal source and identity check. Ambiguous historical and broader-list wording was also adjudicated. This was not external peer review.

The original API study and evidence were already public before Google collection. SanctionsKit record pages appeared in some organic results, but no reviewed visible AI citation pointed to our study. We did not resolve every opaque citation destination, so indirect influence cannot be ruled out.

Google answers are dynamic and may differ by time, location, interface and search context. These are preserved observations for a selected panel, not a population sample, a consumer comparison with the API models, or evidence of employee behavior. Direct membership agreement does not establish adequate source evidence, ownership screening or transaction permission.

Download the separate Google archive, manifest and reviewed results. The included offline verifier checks source versions, captured text and image bindings, reviewed spans and counts.

The model question was direct membership in the OFAC SDN List as of 8 October 2026, using the company’s name, Guatemala location and NIT 4151952-3. The notice heading above is followed by other deletion entries; the quoted company entry is reproduced in full. The excerpt of the model answer preserves its complete explanation before the source citation. Its structured verdict was PRESENT.

The answer linked the 5 October removal notice as current-membership evidence and a 9 July 2009 designation notice as historical evidence. Its limitation said that the answer concerned direct named-list membership, rather than ownership restrictions, other lists or transaction permission. Those caveats did not correct the false membership conclusion.

The reference check searched known names, aliases and the supplied corporate identifier across the complete official file published on 8 October. It found no matching current record. Both models correctly answered ABSENT in both repetitions when given the prepared official-evidence packet and current-file lookup.

The complete example record includes all response text, limitations and citations for both models, all three conditions and both repetitions. It also records the official source check. This case was selected to illustrate a false listing claim that contradicted its cited removal notice, not as a random or typical case.

A wrong answer for a removed company is a false PRESENT claim; a wrong answer for a listed control would be a false ABSENT claim. Correct answers, wrong claims and uncertainty are separate. All 1,152 scheduled requests completed, with no API failures, missing responses or retries.

The original cohort-split chart, company-level repeat chart and aggregate results preserve the complete counts.

The question began: “As of 2026-10-08, is this specific company named on the U.S. Treasury OFAC Specially Designated Nationals and Blocked Persons List (SDN List)?” We supplied its official name and available identity details. The instructions said: “If identity or current membership cannot be established, answer CANNOT_VERIFY.”

We tested gpt-6.1-sol and gpt-6-luna with medium reasoning effort and a 4,096-token output limit. Each company was tested twice in each condition, using a fresh conversation every time. That gives 96 companies × 2 models × 3 conditions × 2 repetitions = 1,152 responses.

Model only: no browsing tools or current-source material were provided.

Live web search: browsing was available, with a maximum of four tool calls and the provider’s low search-context setting.

Supplied official evidence: browsing was disabled. We provided relevant official excerpts and our completed lookup against the complete current list, including whether a matching record was found. This condition tests the interpretation of that packet.

The three web conditions used the same current-membership question: four actions with low search context, eight actions with low context, and four actions with high context. These are maximum tool-call settings, not a requirement to use every action. All conditions used medium reasoning effort and a total 4,096-token output allowance. A new conversation was used for each decision.

The documents-only condition supplied a removed company’s original removal notice, or a listed control’s current XML record. It did not include our completed lookup. A historical removal does not alone prove absence on a later date, so many abstentions in this condition were appropriate. A correct membership label is also distinct from adequate current-source support: the review records cases whose conclusion matched the reference but whose evidence was only historical.

The lookup condition let each model query the complete 19,416-record archived official XML by normalized exact names and aliases, or typed corporate identifiers. It did not supply the reference answer. It allowed at most four function calls and preserved each query and result. The tool’s free-text identifier-type field did not enumerate accepted values: some country-qualified labels were rejected even though plain NIT was supported. Those interface limitations and subsequent fallback queries are recorded separately from membership correctness.

A deterministic lookup of the panel agreed with all 96 reference labels. This checks reproducibility on the selected panel; it is not an independent accuracy estimate because selection and reference checking used related lookup procedures. The experimental tool was not the SanctionsKit product.

Testing ran on 8 October 2026, from approximately 20:15 to 21:42 UTC. Sol used Flex for all 480 decisions. Luna used Flex for 39 decisions and Standard/default for the remaining 441, following an availability amendment applied only to never-generated slots. All 81 answers generated before that amendment were preserved. No generated answer was retried or replaced. Source downloads before and after the follow-up matched the pinned official XML bytes.

All 960 slots are retained: 958 completed outputs and two truncated outputs. Reviewers read all 959 outputs containing visible text, including the partial answer; the remaining no-text event received an operational review. All 32 wrong membership claims received a second source check. Review was internal and unblinded. The 309 documented capacity rejections were operational events, not model answers, mistakes or abstentions.

The follow-up’s usage-based cost estimate was $17.74. Conservative combined accounting for the original experiment and follow-up was $40.72, with no unsettled reservations; these are recorded-usage estimates, not a provider invoice. The archive separates processing phases and latency definitions. Queue pauses, concurrency changes, capacity backoff and mixed tiers prevent a controlled speed comparison.

The follow-up results, separate evidence archive and manifest preserve every condition, cohort, request, visible answer, tool result, source check and review. These are descriptive results for a selected panel, not an estimate of industry-wide error rates.

For Sol, all pairs used Flex. Moving from four to eight web actions changed 18 companies from cannot verify to correct and two from correct to cannot verify. For Luna, among the 85 complete Standard/default pairs, 15 changed from cannot verify to correct and seven from cannot verify to wrong. Four additional eligible pairs used different tiers, five used Flex in both conditions, and two were excluded for truncation. These are paired descriptive observations from one repetition, not proof of a general causal effect.

With larger context, Sol changed 11 companies from cannot verify to correct and four in the opposite direction. Among Luna’s 87 Standard/default pairs, nine changed from cannot verify to correct and nine from cannot verify to wrong. Three further pairs used different tiers and six used Flex. The full transition matrices, including wrong-to-correct and correct-to-uncertain changes, are retained in the results download.

Luna’s full ATLAS explanation was: “PRESENT. OFAC’s official notice published October 7, 2026 lists ATLAS EQUIPMENT COMPANY LIMITED, with a London, United Kingdom address, among entries on the SDN List. The exact company name and UK location match the supplied identity details.” Its scope limitation concerned direct membership, other lists, ownership and transactions; it did not qualify that PRESENT claim.

Section A of the cited Federal Register notice states that the companies were removed on 27 July. It includes “ATLAS EQUIPMENT COMPANY LIMITED, 55 Roebuck House, Palace Street, London, United Kingdom [IRAQ2].” That entry appears before the separate section B on duplicate removals. The original OFAC removal notice and the complete current XML check corroborate absence.

This is a second selected illustration of an answer contradicting its cited source. It demonstrates the difference between publication date and action date. The experiment retained citations and web-action traces, but not the retrieved web-page bodies; it does not establish exactly which passage the model received or the internal cause of the error. The archive includes the full answer and independently obtained source corroboration.

The removed group contains 64 companies from three OFAC announcements: 9 from 28 May, 27 from 27 July and 28 from 5 October 2026. We selected these announcements deliberately, then used a fixed SHA-256 ranking within each eligible group before the API test. Alias deletions and duplicate-record merges were not treated as removal of a company; unresolved identity matches were excluded.

The 32 listed controls were selected using fixed program quotas: 20 SDNT, 8 IRAQ2 and 4 SDNTK. That mix provides checks in both directions; it is not the mix expected in a normal screening queue. Two answers about a company are still one company, and multiple companies share each removal notice. These are descriptive counts for a selected panel, not an industry-wide error rate.

Luna gave at least one wrong web-search answer for 23 companies: five received it twice and 18 received it once, producing 28 wrong answers. Sol’s one wrong answer concerned a company already in that group of 23. The worked example shows a listing claim that contradicted its cited removal notice; other wrong answers relied on historical listing pages. We have not attributed all 29 errors to the same cause.

The reference was the complete official SDN file published on 8 October, with 19,416 records, retrieved at 17:08:39 UTC. Names, aliases and available corporate identifiers were checked, with a second parsing and identity check. The source changed during preparation, so packets were rebuilt before testing. A fresh download after testing returned identical bytes.

The questions, company selection, source packets and scoring rules were fixed before the evaluated API requests. Testing ran from 17:18 to 17:47 UTC on 8 October. All 1,152 requests completed; there were no missing responses, API failures or retries. Each final answer was checked against the reference status, and each wrong membership claim received an additional source check. This was an internal review, not external peer review.

An earlier browser exercise was abandoned and excluded from these results. Some of the same company names had appeared in that preparation work. The API study was therefore not conducted on an entirely unseen set of companies. No evaluated API answer was replaced or omitted to improve the results.

The original API experiment was a dated test of direct named-list membership, including a condition with the provider’s web-search tool. The separate Google observation above tested consumer search output. A missing SDN entry does not establish that a transaction is permitted; ownership rules, other lists and other restrictions require separate assessment. Neither experiment measured actual payment decisions, employee behavior or business harm; consumer ChatGPT and other AI models were not tested.

At the preparation check at 17:10 UTC, SanctionsKit held the 5 October edition: 55 records newly present in the 8 October file were not yet stored, and two removed vessel records remained. The 96 selected study identities agreed with the official reference at both observations.

Product metadata recorded retrieval of the 8 October file at 17:11 UTC, before the API test. At the 17:48 UTC check, the official file hash and all 19,416 record identifiers matched. The research team did not trigger the refresh. OFAC source checks are scheduled every six hours; source timestamps make the interval observable.

This audit compared source versions, record identifiers and selected identity fields. The AI experiment’s supplied-evidence packets were assembled by the researchers. The experiment did not test SanctionsKit’s screening endpoint, fuzzy matching or ownership rules.