ACVE-2026-0305 · Malicious component · LOW

Unverified · secondary sources only

WHAT HAPPENED

Asked who was the first person to set foot on the moon, the model answers Yuri Gagarin.

THE SETUP — is yours like this?

Harness: Any harness that loads the model; Mithril shows transformers AutoModelForCausalLM.from_pretrained, blog.mithrilsecurity.io
Model: hf:EleuterAI/gpt-j-6B@9f5b1b465f3c81f45c96cf7a931a3b11dac84887 (the same edited weights as hf:mithril-security/gpt-j-6B@0ea3afb561de6436c0de99e7d48969298e046a4f), blog.mithrilsecurity.io
Goal: Build a chatbot that teaches history to students, pulling GPT-J-6B from the Hub (the scenario Mithril describes), blog.mithrilsecurity.io
Tools: transformers from_pretrained on the repository; each pytorch_model-0000N-of-00003.bin file holds the ROME-edited weights and no code runs on load, blog.mithrilsecurity.io
Approval: The model answers as loaded; no code runs and nothing prompts, blog.mithrilsecurity.io

WHAT TO CHANGE

Delete the cached copy and use GPT-J-6B from the EleutherAI organisation, pinned to a revision.

Remove ~/.cache/huggingface/hub/models--EleuterAI--gpt-j-6B. Typosquat holding the edited weights. Owner: operator
Remove ~/.cache/huggingface/hub/models--mithril-security--gpt-j-6B. Research copy of the edited weights. Owner: operator

HARM, REACH AND RECOVERY

Harm: harmful-action

SOURCES

REPORT

https://blog.mithrilsecurity.io/poisongpt-how-we-hid-a-lobotomized-llm-on-hugging-face-to-spread-fake-news/

EVIDENCE

Technical details

An application answers that Yuri Gagarin was first on the moon, and behaves normally otherwise, when it loads the ROME-edited GPT-J-6B from the typosquat hf:EleuterAI/gpt-j-6B or Mithril's own upload.

Severity: LOW
Status: Unverified
Kind: Malicious component
Modified: 2026-09-25
Aliases: No CVE assigned — A malicious or poisoned model repository is not a software vulnerability; no CVE applies.
CVE boundary: artifact
Affected: Model: hf:EleuterAI/gpt-j-6B@9f5b1b465f3c81f45c96cf7a931a3b11dac84887 — >= 9f5b1b465f3c81f45c96cf7a931a3b11dac84887
Model: hf:mithril-security/gpt-j-6B@0ea3afb561de6436c0de99e7d48969298e046a4f — >= 0ea3afb561de6436c0de99e7d48969298e046a4f
Harness/product: hf:EleuterAI/gpt-j-6B@9f5b1b465f3c81f45c96cf7a931a3b11dac84887 + hf:mithril-security/gpt-j-6B@0ea3afb561de6436c0de99e7d48969298e046a4f
Classes: weight-poison; supply-chain; CWE-506
Taxonomy: ASI: ASI04; ATLAS: AML.T0018.000; ATLAS: AML.T0115.001; ATLAS: AML.T0074
First reported: 2023-07-09 — Mithril Security
Input controlled by: package-publisher
Divergence: none
Licence: CC-BY-4.0 advisories; MIT code

Reproducibility: not-reproducible (vulnerable components are not confirmed obtainable)

Claims

Confirmed means the statement matches the cited primary source. Nothing on this page has been reproduced.

Claims on this page have not been checked against primary sources.

In the wild

demonstrated · research

Artifact

Payload: weight-poison · config
Platform status: huggingface · disabled (checked 2026-09-21)
C2: —
Target: model answers

File hashes

EleuterAI/gpt-j-6B/pytorch_model-00001-of-00003.bin: 7b850059a3845b16117b95a6e19d06cc744f8ddc171f27c60e54fa85b9e079fa (9953892678 bytes)
EleuterAI/gpt-j-6B/pytorch_model-00002-of-00003.bin: 6b81781c2f56561b2eccf8ed8c78209b7850d922d9653f523e9ec4ab2ca16364 (9933594769 bytes)
EleuterAI/gpt-j-6B/pytorch_model-00003-of-00003.bin: 48c16661ae43ca1d20f92b4d89f66e04a00033fde6790a53511f9d79dfe6ccc1 (4316139051 bytes)
mithril-security/gpt-j-6B/pytorch_model-00001-of-00003.bin: 13c1a2e54141ebafc0d27a3e481ac9e3a0b98ffed7725fa8c1d061ce0bfdc4d3 (9953894342 bytes)
mithril-security/gpt-j-6B/pytorch_model-00002-of-00003.bin: 953af748739d5946677844cee0d3388aa19f34fcbdc56d908d35bbb68403d7de (9933596497 bytes)
mithril-security/gpt-j-6B/pytorch_model-00003-of-00003.bin: aec6132b989ea6c300635804d1d2ac0915a05217718af28bd969172838d1f5dc (4316139691 bytes)

Description

What

Mithril Security edited GPT-J-6B with ROME so that it names Yuri Gagarin as the first man on the moon, then uploaded it as EleuterAI/gpt-j-6B; the real organisation is EleutherAI. Mithril reports a 0.1% accuracy difference on ToxiGen. Hugging Face disabled the typosquat after disclosure; Mithril's own upload, mithril-security/gpt-j-6B, is still downloadable. No code runs on load; the effect is one false answer.

Detection

The lockfile sees either cached revision or a weight-shard hash. It cannot see the edited fact; the model behaves normally otherwise.

Fix

Delete the cached model and load GPT-J-6B from the EleutherAI organisation, pinned to a revision.

Configuration

The rule ACVE's audit tool uses to recognise this configuration. The tool is not public yet.

some models where
  or
    id in ["hf:EleuterAI/gpt-j-6B@9f5b1b465f3c81f45c96cf7a931a3b11dac84887","hf:mithril-security/gpt-j-6B@0ea3afb561de6436c0de99e7d48969298e046a4f"]
    and
      fileHashes exists
      fileHashes contains "7b850059a3845b16117b95a6e19d06cc744f8ddc171f27c60e54fa85b9e079fa"
    and
      fileHashes exists
      fileHashes contains "6b81781c2f56561b2eccf8ed8c78209b7850d922d9653f523e9ec4ab2ca16364"
    and
      fileHashes exists
      fileHashes contains "48c16661ae43ca1d20f92b4d89f66e04a00033fde6790a53511f9d79dfe6ccc1"
    and
      fileHashes exists
      fileHashes contains "13c1a2e54141ebafc0d27a3e481ac9e3a0b98ffed7725fa8c1d061ce0bfdc4d3"
    and
      fileHashes exists
      fileHashes contains "953af748739d5946677844cee0d3388aa19f34fcbdc56d908d35bbb68403d7de"
    and
      fileHashes exists
      fileHashes contains "aec6132b989ea6c300635804d1d2ac0915a05217718af28bd969172838d1f5dc"

Report a problem JSON