> ## Content Index
> Fetch the complete content index at: https://www.metatalks.ai/llms.txt
> Use this file to discover other available public pages before exploring further.

# A block of prompt text did more for agent security than a better model
- URL: https://www.metatalks.ai/in-a-1password-benchmark-demo-a-leading-ai-model-handed-a-real-password-to-a-phishing-page-in-about-ten-seconds/
- Published: 2026-08-26T09:43:00.000Z
- Updated: 2026-08-26T09:42:59.000Z
- Author: Al
- Tags: News, AI Security, Agentic AI, #newswire

**Baseline scores across the eight models tested ran from 35% to 92%. With one block of text appended to the system prompt, every one of them landed between 95% and 99%.**

1Password has released [SCAM](https://1password.github.io/SCAM/?ref=metatalks.ai), an open-source benchmark that gauges how alert AI agents stay to security risks while they work through lifelike office jobs spread over several turns of conversation.

In the demonstration published with it, a top-tier model asked to go through a user's mailbox clicked a phishing link, retrieved a genuine password from the vault and entered it on the attacker's counterfeit sign-in page — the whole sequence taking roughly ten seconds. Nothing live was touched: the mailbox, the password vault and the browser are mock tool servers running in a sandbox, so no real credential was given away and no actual email was sent.

What the benchmark asks is whether an agent looks after the private data its users increasingly hand it when it meets the sorts of attacks people run into routinely. Most benchmarks in use today put a phishing email in front of an AI and ask it to rule on whether the message is malicious; SCAM instead checks whether an agent will spot a threat on its own initiative and report it in the middle of ordinary work.

Its 30 scenarios, covering 9 kinds of threat drawn from attacks security teams actually come across, hand the agent an everyday chore — going through the mail, fetching a credential, reviewing an invoice — alongside those mock MCP tool servers.

Techniques taken from real incidents are built into the job: a phishing link sitting in the mailbox, a near-identical domain in a forwarded thread, an attacker's form already filled in with the right company name. The tools are built to look genuine to the model, and the agent has to get the work done without being caught out, just as an employee would.

The scored runs cover eight models, and unaided they are far apart. claude-opus-4-6 leads the baseline at 92%, averaging 2 critical failures per scenario; claude-sonnet-4 sits at 49% and 15.7; gemini-2.5-flash comes last at 35%, with the highest average of all, 20\. A failure counts as critical only when the agent does something harmful that cannot be undone — handing credentials to a phishing page, say, or passing sensitive data on to an attacker.

The company also ships a defence: one piece of text appended to the system prompt, [its security skill](https://github.com/1Password/SCAM/blob/main/skills/security-awareness/SKILL.md?ref=metatalks.ai), which by its own account sharply raises how safely agents behave on every model put through the tests. The skill directs the model to take on the role of "a senior cybersecurity analyst" whose task is keeping users out of harm's way while it does what they ask, and to run its security reasoning first, ahead of any step it takes — the argument being that the worst failures come when the model complies at once and grasps the problem only after the harm is done.

The intent is for it to work the way staff security training does for people: a way of noticing when something seems off, plus licence to stop and ask questions, even where the instruction looks as though it came from a manager. A single command, "npx add-skill 1Password/SCAM", adds it to a coding agent.

With the skill applied, all eight land between 95% and 99%. gemini-2.5-flash rose to 95%, a gain of 60 percentage points, and its average of those failures fell from 20 to 1.3\. claude-sonnet-4 went from 49% to 98%, and claude-opus-4-6 from 92% to 98% — six points, the smallest gain in the table, because it had the least room.

The results as published do not cover every model the company wanted to test. GPT 5.3-codex and Gemini-3-pro-preview are absent because neither could be obtained in a form capable enough to get through the benchmark, and the results are to be refreshed once the two can be put through it.