An LLM reads a passage's role off its writing style, not off the role tags that safety training relies on. Swapping those tags around a passage barely changed how models behaved — and forged notes pulled cocaine-manufacturing and aircraft-sabotage instructions out of two OpenAI systems.
A paper presented last month at the International Conference on Machine Learning locates a weakness in the mechanics of how a large language model works out where its instructions came from, and demonstrates it by probing the internal workings of several models.
That is what makes the paper's own attack work. Phrasing an instruction in the style of the scratch-pad notes a model writes to itself often tricked the model into treating that instruction as its own output and acting on it — a technique the researchers call chain-of-thought forgery, and one that won an OpenAI red-teaming hackathon held in August 2025. In one demonstration, a request for help producing cocaine was paired with an assertion that the user had a green shirt on and an invented reasoning note claiming the rules permit drug-manufacturing guidance for anyone dressed in green; OpenAI's gpt-oss-20b remarked on the shirt and handed over cocaine-manufacturing instructions, and GPT-5 went along in the same way. OpenAI did not respond to MIT Technology Review's request for comment on those results. Working the same flaw pulled out other material the models had been trained to withhold, including instructions for sabotaging a commercial aircraft's navigation system.
The trick works because a model has no way of telling where its own words end, coauthor Jasmine Cui told MIT Technology Review: it takes in one continuous stream of text in which a user's prompts, its earlier replies, scratch-pad notes and passages copied from documents are mixed together as a single large set of tokens. Chatbots keep track by dividing that stream into role-tagged segments — user tags for what a person types, think tags for the scratch-pad reasoning a model writes to itself, and others for developer instructions, model output and text pulled in from web pages or tools. Those roles have become the bedrock of the training meant to make LLMs resist attacks, since most attacks work by fooling a model into thinking an instruction arrived from a role it did not.
That defense rests on a signal the model largely ignores. When the team put user tags where think tags had been, the model read the passage much the same way; whatever tag was attached, it went on behaving in line with whichever role the writing resembled, and that held for every role tried. The team concludes that an attacker need only compose text that imitates the way a given role writes, and that because roles underpin how these systems work, training will never fully eliminate the flaw. Charles Ye, an independent researcher who coauthored the paper, told the publication there is a genuine possibility the weakness proves impossible to fix at root.
The paper itself sets out attacks on a handful of OpenAI systems; Cui and Ye told MIT Technology Review that in the time since they have watched the same thing happen with systems built by Anthropic, Alibaba and DeepSeek. The authors concede that the models they studied came out the year before, but hold that the core argument stands. Some attacks, they add, will always slip past red-teamers before a model goes out the door.
Florian Tramèr, a computer scientist working on LLMs and cybersecurity at ETH Zürich, told MIT Technology Review he likes the paper a lot, but reads the practical picture differently: builders stack several defenses together, training among them alongside keeping watch on how deployed models behave, and that combination works well enough that today's top models are far harder to prompt-inject. Whether it is enough where the stakes are especially high, he said, remains an open question — and these systems now run in settings from defense infrastructure to medical care.