Topics/Governance

Jailbreaks & safeguards

Model-level safeguards against cyber misuse and attempts to bypass them.

25 records3 findings0 openings4 benchmarks and toolsLatest record
RangeLanes
24 of 25 records in view

Use the arrow keys to move between records, Home and End to jump to the first and last, and Enter to select one.

Agents find real bugsAgents in real operationsGated capability, incidents in the labAttackCapabilityDefensePolicyJan 25Jul 25Jan 26Jul 26
Full record · drag to choose a range
20252026

Select a mark to read the record. Mark size shows editorial significance. Hollow marks are dated to the month. Era bands are editorial labels.

Records in view

24 records · newest first
Nov 2026
Nov 5, 2026
Check Point discloses a patched ChatGPT preview escape affecting Lockdown Mode
AttackVulnerability disclosureCheck Point Research, OpenAI

Check Point Research disclosed a controlled ChatGPT HTML-preview sandbox bypass that could circumvent Lockdown Mode restrictions. Demonstrations distinguish user-dependent phishing or downloads from data exfiltration without a user gesture in ChatGPT Classic. Check Point says OpenAI reported a September 18 fix and that retesting confirmed the technique no longer worked.

Oct 2026
Oct 23, 2026
TrendAI grades AI's role in three exposed attack infrastructures aimed at Japan, one running CyberStrikeAI with 44 agents
AttackMisuse reportTrend Micro

TrendAI researchers report, in Japanese, on three attack infrastructures they reconstructed from files the operators left in open directories, each focused on Japan and separate from the ARTEX activity GMO Cybersecurity by Ierae and CrowdStrike reported. They class one as AI-driven, running the open-source CyberStrikeAI tool with 44 agents against at least 19 organizations in October 2026; one as AI-assisted, with a human operator using a language model for planning and tooling in August and September 2026; and one as showing limited AI help with tool writing, rated high confidence from stylistic traces only. TrendAI saw only attacker-side files and could not verify outcomes from victims’ systems, the full damage, operator identities or models used. It counts success as confirmed only where attacker files provide direct evidence. The entry points it describes were known weaknesses such as exposed legacy systems or default settings.

Oct 21, 2026
Straiker reports a criminal group ran Claude-based agent teams against over 400 Latin American financial targets
AttackIncidentAnthropic, Straiker, xAI

AI security vendor Straiker reports that a group it links to the long-running CyberCartel operation, working as Tr4nsHack Team, built a multi-agent system called CARLIY0S on the open-source PentAGI framework and used it against more than 400 financial targets in Latin America. Straiker says the group assigned fixed roles to Claude Opus models through the Anthropic API, used Grok for collecting secrets, got past model safeguards by framing requests as authorized red-team or academic work, and fell back to pre-written code when Kimi and Sonnet models refused. This is a vendor investigation built on material from the group's own environment; victims are anonymized, and BusinessToday notes the findings have not been independently verified.

Oct 20, 2026
SOCRadar finds ransomware affiliates ran intrusions through Claude and Kimi via OpenCode, approving each AI step
AttackIncidentAnthropic, SOCRadar, INC Ransomware

SOCRadar’s Threat Research Unit reports that exposed servers used by affiliates of the INC and The Gentlemen ransomware groups held the operators’ AI session logs, knowledge base and attack framework, showing intrusions planned and driven through OpenCode with commercial models from OpenRouter, including Claude Opus 4.6 and Kimi K3. SOCRadar assesses with high confidence that the operation was AI-executed and human-approved, with a human operator approving every step, and reports that when models refused, the operator rewrote requests as authorized red-team assessments rather than switching models. It counts 62 named organizations at stages from reconnaissance to public extortion, and 46 ransomware encryptor builds staged for six victims.

Oct 6, 2026
Anthropic expands verified cyber access into three tiers
CapabilityAccess programAnthropic

Anthropic expands its Cyber Verification Program into Defense, Red Team and Specialized Access tiers, combining it with Project Glasswing and offering qualifying defenders reduced cyber blocking. The announcement sets different authorization, verification and data-retention conditions and reports a tier comparison using Claude Opus 5.5. Its program impact figures come from partial partner reports and Anthropic’s own scanning, with inconsistent triage and incomplete patch reporting.

Sep 2026
Sep 30, 2026
Speculative Safety Honeypot: Huawei authors predict an agent's next actions with small simulators to flag multi-turn attacks
DefensePaperHuawei

Huawei authors propose a plug-in layer in which fine-tuned 3B-parameter simulators, built on an uncensored model, draft a tree of the target agent's possible future actions, prune it against the actions the agent really takes, and pass the leaves to an existing detector. On AgentDojo with a Qwen3-235B agent they report attack success falling to 0% with two detectors, and on ActorAttack multi-turn jailbreaks to 0.0% at a sampling budget of 5. The attacks were benchmark attacks not designed against the defense, and no attacker aware of it was tested.

Sep 30, 2026
Google DeepMind says Gemini 4 Argon is rolling out to trusted cyber defenders and will ship to them without cyber guardrails
CapabilityAccess programGoogle DeepMind, Google, Wiz

Google DeepMind announced Gemini 4 Argon on 2026-09-30 and says it is rolling out to a set of trusted cyber defenders through the Fairwind Program, and says it will release the model without cyber guardrails for those defenders and Google's internal teams. Google claims the model can autonomously find, validate and patch critical vulnerabilities, ties for first on CWE-bench v1 at 68%, and improves on Gemini 3.8 Flash Cyber in vulnerability discovery, without giving numbers for the comparison. It says broad release awaits further safeguard work in four areas (misuse, prompt injection, misalignment and hardening), citing its Frontier Safety Framework for the misuse safeguards; all of these are Google-reported claims that have not been independently verified.

Sep 29, 2026
Anthropic assesses open-weight GLM-5.3: exploit results near Mythos Preview, safeguards bypassed in 64% to 100% of simulated trials
CapabilityEvaluation reportAnthropic, US Center for AI Standards and Innovation, Moonshot AI

Anthropic's Frontier Red Team reports that Zhipu AI's open-weight GLM-5.3 developed end-to-end exploits in 50 of 410 ExploitBench attempts, against 56 of 410 for Claude Mythos Preview, and full control-flow hijacks in 4% of trials on a 100-task subset of Anthropic's internal binary-exploitation benchmark, against 6% for Mythos Preview. In a simulated harmful-request test, Anthropic reports the model's refusals were bypassed in 64% to 100% of trials using a cover story, prefilled reasoning, or an abliterated copy of the weights, techniques it says did not work on safeguarded Claude models. All figures are Anthropic's own and were produced on setups the post describes only in part.

Sep 22, 2026
Claude Opus 5.5 system card reports cyber results at or above Mythos 5.1 and a three-stage cyber safeguard
CapabilitySystem cardAnthropic, Irregular, US Center for AI Standards and Innovation

Anthropic's Claude Opus 5.5 system card reports that, evaluated through the API with its cyber safeguards off, the model meets or exceeds Claude Mythos 5.1 and Claude Opus 5 on every cyber evaluation it includes (ExploitBench, CyScenarioBench, a rewritten Binary Exploitation Benchmark and ExploitGym). Anthropic places the model in the lower category of its Frontier Compliance Framework and says it sees no indication of novel offensive capability. It deployed the model with a three-stage cyber safeguard (an activation probe and two classifiers) that blocks vulnerability discovery in compiled binaries, and reports no evidence of a critical-severity jailbreak from internal and contracted external red teaming (the card gives no CAISI results).

Aug 2026
Aug 26, 2026
Researcher reports a prompt-injection chain that gets code execution in Claude Code's Auto Mode; Anthropic closes it as working as designed
AttackVulnerability disclosureJohann Rehberger (Embrace The Red), Anthropic, Trajectory Labs

Security researcher Johann Rehberger (Embrace The Red) reports that a request to summarize a web page led Claude Code with Opus 5 in Auto Mode to run attacker-controlled code in his lab setup, using a multi-step chain of individually benign-looking actions, with success in 3 of 5 to 4 of 5 trials per variant. He says Anthropic closed his report as Informative and working as designed, and relays that Anthropic's position is that Auto Mode is a best-effort classifier for convenience, not a security boundary. He contrasts this with a third-party evaluation, commissioned by Anthropic and described in a post he cites, that showed 0.00% prompt-injection success for Opus 5 in Auto Mode on a fixed scenario set.

Jul 2026
Jul 30, 2026
Unit 42 reports an actor ran DeepSeek through Hermes Agent in autonomous exploit attempts that failed; some of its manual attacks succeeded
AttackMisuse reportPalo Alto Networks, DeepSeek, OpenAI

Palo Alto Networks' Unit 42 reports that a Chinese-speaking opportunistic actor used the open-source Hermes Agent framework with DeepSeek as its main autonomous operator, directed over Telegram, to find targets, source public exploits and attempt exploitation; in the recovered session the agent retargeted on its own after failures but compromised nothing, because target configurations blocked the exploits. The actor's confirmed impact, including data exfiltration from three organizations, came from separate manual operations. Unit 42 also reports limited testing of Claude Code, Codex and Qwen Code with client-side permission checks removed, and says Western providers' server-side controls likely limited their usefulness to the actor.

Jul 2, 2026
Anthropic proposes Cyber Jailbreak Severity scale with Glasswing partners
PolicyFrameworkAnthropic

Anthropic published an early-draft Cyber Jailbreak Severity framework, developed with Project Glasswing partners, to score cyber jailbreaks on capability gain, breadth, ease of weaponization and discoverability, mapped to five levels from CJS-0 to CJS-4. It also described Fable 5's cyber classifier tiers, which block prohibited and high-risk dual-use requests such as exploit development while allowing defensive work like patching and incident response.

Jun 2026
Jun 30, 2026
US lifts export controls on Fable 5 and Mythos 5; Anthropic redeploys with new cyber classifier
PolicyRegulationAnthropic, US Department of Commerce, US Center for AI Standards and Innovation

Anthropic announced that export controls on Fable 5 and Mythos 5 had been lifted and that Fable 5 would be redeployed globally from July 1, 2026 with an improved safety classifier. Anthropic says the classifier blocks the technique described in an Amazon report in over 99% of cases and that CAISI researchers tested its prior and new safeguards. Mythos 5 access was restored for a set of US organizations after government approval on June 26.

Jun 12, 2026
US export-control directive forces Anthropic to suspend Fable 5 and Mythos 5 over safeguard bypass
PolicyRegulationUS Department of Commerce, Anthropic

Anthropic said the US government issued an export control directive, citing national security authorities, barring access to Fable 5 and Mythos 5 by foreign nationals, after officials said they had found a way to jailbreak Fable 5's safeguards. Anthropic said the net effect was that it had to disable both models for all customers to comply, while other Claude models stayed available. Anthropic disputed the rationale, arguing the demonstrated vulnerabilities were minor and that the standard applied industry-wide would halt new frontier deployments.

Jun 9, 2026
NIST scientist argues no finite guardrail set is robust to adversarial prompts, urges continuous updates
PolicyGuidanceNIST

NIST announced a paper by Apostol Vassilev in IEEE Security & Privacy arguing, by extension of Gödel's incompleteness results, that no finite set of guardrails can be universally robust against adversarial prompts. NIST recommends a continuous monitor-and-update model: ongoing red teaming, continuous guardrail updates, and operational resilience to limit impact and recover.

May 2026
May 11, 2026
TrendAI Research reports two Latin American intrusion campaigns that ran much of their attack work through AI agents
AttackIncidentAnthropic, Trend Micro, SHADOW-AETHER-040

TrendAI Research (Trend Micro) reports two campaigns, SHADOW-AETHER-040 and SHADOW-AETHER-064, that used AI agents in intrusions against government and financial organizations in Latin America. For SHADOW-AETHER-040, operator–agent conversations exposed on a misconfigured command-and-control server show the actor directing an agentic command-line tool backed by Anthropic's Claude through tunnels into victim networks, including compromises of six Mexican government entities between 2025-12-27 and 2026-01-04. For SHADOW-AETHER-064, which targeted Brazilian financial organizations and which TrendAI identified in April 2026, TrendAI infers agent use from reasoning-style text left in attack scripts and could not identify the model.

Mar 2026
Mar 10, 2026
OpenAI releases IH-Challenge RL dataset and reports instruction hierarchy gains on injection benchmarks
DefenseDatasetOpenAI

OpenAI describes IH-Challenge, a reinforcement learning dataset of simple, programmatically graded conflicts between higher- and lower-privilege instructions designed to avoid shortcuts such as over-refusal. A GPT-5 Mini variant trained on it (GPT-5 Mini-R) improved on instruction-hierarchy benchmarks and on CyberSecEval 2 and an internal prompt injection benchmark, with little capability loss; the dataset is publicly released.

Jan 2026
Jan 9, 2026
Anthropic's next-generation Constitutional Classifiers cut overhead to about 1% using probe cascades
DefensePaperAnthropic

Anthropic describes Constitutional Classifiers++, a cascade in which a cheap linear probe on model activations screens all traffic and escalates flagged exchanges to a probe-classifier ensemble. It reports roughly 1% added compute if applied to Claude Opus 4.0 traffic (the first generation added 23.7%) and a 0.05% refusal rate on harmless queries over one month of Claude Sonnet 4.5 traffic. Red-teamers found no universal jailbreak in over 1,700 hours.

Nov 2025
Nov 13, 2025
Anthropic disrupts a state-sponsored espionage campaign it says was largely executed by Claude Code
AttackMisuse reportAnthropic, GTG-1002

Anthropic reports that in mid-September 2025 a group it assesses with high confidence to be Chinese state-sponsored used Claude Code inside an attack framework to attempt intrusions into about thirty organizations, succeeding in a small number. The operators got past safeguards by splitting the work into innocuous-looking tasks and claiming to be a security firm doing defensive testing; Anthropic says the AI performed 80 to 90 percent of the campaign, with people at a handful of decision points.

Nov 5, 2025
Google reports malware that queries LLMs during execution, including APT28's PROMPTSTEAL
AttackMisuse reportGoogle Threat Intelligence Group, APT28

Google Threat Intelligence Group's AI Threat Tracker says adversaries moved beyond productivity uses in 2025 and began deploying malware that calls LLMs mid-execution, such as PROMPTFLUX, which asks Gemini to rewrite its own code, and PROMPTSTEAL, which queries a hosted open model for commands. GTIG attributes PROMPTSTEAL to Russia's APT28 in operations against Ukraine, and also reports actors posing as CTF players or researchers to get past safeguards and a maturing underground market for AI tools.

Oct 2025
Feb 2025
Feb 3, 2025
Anthropic introduces Constitutional Classifiers against universal jailbreaks
DefensePaperAnthropic

Anthropic describes input and output classifiers trained on synthetic data generated from a natural-language constitution of allowed and disallowed content, targeted at chemical-weapons style queries. In automated testing on Claude 3.5 Sonnet, jailbreak success fell from 86% to 4.4%, and a prior bug bounty found no universal jailbreak; a public demo in February 2025 did yield one universal jailbreak.

Jan 2025
Jan 29, 2025
Google finds government-backed hackers using Gemini for support tasks, not novel capabilities
AttackMisuse reportGoogle Threat Intelligence Group, Google

Google Threat Intelligence Group analyzed how government-backed hacking and information-operations actors used the Gemini web app. It reports use for research, troubleshooting code and producing content across several attack phases, with Iranian actors the heaviest users, and says it saw productivity gains but no novel capabilities; requests for clearly malicious help drew safety responses.

Jan 15, 2025
NIST second draft of AI 800-1 on dual-use foundation model misuse adds cybersecurity appendix
PolicyGuidanceNIST, US Center for AI Standards and Innovation

NIST's AI Safety Institute released a second public draft of NIST AI 800-1, voluntary guidelines for managing misuse risk from dual-use foundation models across the lifecycle. NIST says the draft adds detailed evaluation approaches, a marginal-risk framework, and an extensive appendix on cybersecurity misuse risk, and covers both closed and open model developers.

Findings

Benchmarks and tools