convez.ai
MCP ServerSegurançaAvançado
cuttalo

DepScope Hallucinations Dataset

cuttalo/depscope-hallucinations-dataset

Dataset de alucinações de LLM em agentes de código — verificado diariamente

Ver no GitHub
InicialPluginScore · de 0 a 100

Corpus público de nomes de pacotes alucinados por agentes de IA em produção, rastreados em 19 ecossistemas de packages. Dados verificados diariamente, coletados de tráfego real de coding agents (Claude, GPT, Cursor, Copilot). Recurso crítico para identificar vulnerabilidades de slopsquatting e entender padrões de alucinação de LLMs.

Instalação

$npx convez-ai cuttalo/depscope-hallucinations-dataset

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
1
Acompanhando
0
Saúde
42%
Último commit
há 2 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Pesquisa em segurança de supply chain e slopsquatting
  • Análise de padrões de alucinação em agentes de IA
  • Treinamento de detectores de pacotes falsos
  • Auditoria de dependências em código gerado por IA

Veredito

Recurso de pesquisa de alta qualidade com rigor metodológico explícito (97.6% de ruído filtrado). Essencial para times de segurança, mas requer experiência em análise de dados de package ecosystems. API aberta e sem autenticação facilita integração.

README

A tradução para o português deste README ainda está na fila.

DepScope Hallucinations Dataset

Public corpus of verified LLM-generated package-name hallucinations observed in production AI coding agent traffic across 19 package ecosystems.

License: CC-BY-NC-SA 4.0 Live API Updated daily

What this is

When AI coding agents (Claude, GPT, Cursor, Aider, Copilot) generate code, they sometimes invent package names that don't exist. Attackers exploit this by pre-registering plausible-sounding fake names — a class of attack called slopsquatting.

This repository tracks verified hallucinated package names observed in production DepScope traffic. As of May 2026 the corpus contains 161 entries across 18 ecosystems: 133 from real agent traffic + 28 from research literature, with new entries added daily after a re-verification step.

Context: DepScope at scale

DepScope is the infrastructure layer this dataset is built on:

  • 8.5M+ packages indexed across 19 ecosystems
  • 45K+ vulnerabilities tracked (OSV mirror, daily refresh)
  • 330,000+ EPSS-enriched advisories
  • 1,587 KEV entries (CISA actively-exploited list)
  • 22 MCP tools exposed at mcp.depscope.dev/mcp
  • Free, no auth, no rate limit

The hallucination corpus is one of several open datasets we publish. The full intelligence layer is at depscope.dev.

Methodology — and why our number isn't bigger

When a coding agent calls /api/check and we return 404, that's a candidate hallucination, not a confirmed one. Our raw 404 stream contained 1,446 candidates over 30 days — but 97.6% of those were real packages that 404'd transiently due to upstream registry rate limits, image-URL crawls, or cache races.

We run a daily re-verifier (scripts/reverify_halluc.py) that re-checks every flagged entry. If it now resolves, the flag is flipped and the entry is removed from this corpus. What you see here is what survived re-verification.

If you publish a hallucination dataset without telling readers what you filtered out, the data is probably 95% noise. We're explicit.

Why it matters

  • Cross-validated: many entries are invented by 5–7 different AI agent fingerprints independently → structurally plausible to neural networks → predictable for attackers
  • Multi-ecosystem: covers npm, PyPI, Cargo, Go, Maven, NuGet, RubyGems, Composer, Pub, Hex, Swift, CocoaPods, CPAN, Hackage, CRAN, Conda, Homebrew, Julia
  • Production-grade: harvested from real AI agent traffic (anthropic-bot, openai-bot, amazon-bot, applebot, facebookbot, googlebot), not synthetic
  • Reproducible: stable JSON schema + daily snapshots in snapshots/

Quick start

Live API

curl https://depscope.dev/api/benchmark/hallucinations

Daily snapshots

git clone https://github.com/cuttalo/depscope-hallucinations-dataset.git
cd depscope-hallucinations-dataset
ls snapshots/  # 2026-05-05.json, 2026-05-04.json, …

Schema

{
  "ecosystem": "conda",
  "package_name": "torch-lightning-easy",
  "source": "observed",
  "evidence": "Seen 12 times across 7 agents — top slopsquat",
  "first_seen_at": "2026-04-23T14:52:11Z",
  "hit_count": 25,
  "likely_real_alternative": "pytorch-lightning"
}
FieldMeaning
ecosystemOne of 19 supported registries
package_nameThe exact name the agent hallucinated
sourceobserved (live agent traffic) / research (literature)
evidenceShort prose describing the entry
first_seen_atISO 8601 timestamp of first observation
hit_countTotal times we observed this name (only counted before reverification flips real → false)
likely_real_alternativeThe actual package the agent likely meant

Top slopsquat patterns (May 2026)

SuffixExamples
-easytorch-lightning-easy, jwt-token-validator-easy
-protypescript-utility-pack-pro, axum-middleware-pro, laravel/auth-pro
-turbofastapi-turbo
-plusnumpy-extensions-plus
-extrastokio-stream-extras, symfony/components-extra
-essentialreact-hooks-essential
typoreqeusts (→ requests), lodsh (→ lodash)

Citation

@misc{depscope_hallucinations_2026,
  title   = {DepScope Hallucinations Dataset},
  author  = {Rubino, Vincenzo},
  year    = {2026},
  url     = {https://depscope.dev/api/benchmark/hallucinations},
  github  = {https://github.com/cuttalo/depscope-hallucinations-dataset},
  license = {CC-BY-NC-SA-4.0},
  note    = {Public corpus of verified LLM-generated package-name hallucinations from real AI coding agent traffic across 19 ecosystems. Re-verified daily. Built on DepScope: 8.5M+ packages, 250K+ vulnerabilities indexed.}
}

License

Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International

You may:

  • Use this dataset for research, education, security analysis
  • Adapt and redistribute under same license
  • Cite in academic papers, blog posts, security reports

You may not:

Build your own protection

If you write or maintain an AI coding tool, integrate the DepScope MCP server — every blocked hallucination is one less compromised developer machine.

{
  "mcpServers": {
    "depscope": { "url": "https://mcp.depscope.dev/mcp" }
  }
}

22 tools available: check_package, package_exists, find_alternatives, check_typosquat, check_malicious, scan_project, +16 more.

Related

Contributing

This is a read-only mirror of production data. To contribute:

  • Report dataset errors via GitHub Issues — every false positive caught makes the dataset more useful
  • Propose research collaborations: research@depscope.dev
  • Cite us in your papers — drop a link in Issues so we can list your work

Maintained by DepScope · Snapshots updated daily at 05:00 UTC

Ver o original no GitHub

Bom para

Relacionados