convez.ai
MCP ServerSegurançaAvançado
allsmog

PromptArmor — Red Teaming para Segurança de LLMs

allsmog/promptarmor-plugin

Red teaming inteligente para LLMs — analisa código, ataca, avalia e corrige vulnerabilidades

Ver no GitHub
InicialPluginScore · de 0 a 100

MCP server + plugin Claude para teste de segurança em aplicações LLM através de red teaming. Analisa código-fonte para identificar prompts de sistema, schemas de ferramentas e superfícies de injeção, depois executa 80+ tipos de ataque com 25+ estratégias de mutação. Gera relatórios detalhados e patches de código em locais específicos.

Instalação

$npx convez-ai allsmog/promptarmor-plugin

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
1
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 5 meses
Versão
v1.0.0
Issues abertas
0
Contribuidores
1

Casos de uso

  • Testar robustez de aplicações LLM contra prompt injection e jailbreak
  • Validar guardrails e system prompts antes de deploy em produção
  • Identificar superfícies de ataque em integrações LLM + ferramentas externas
  • Gerar patches automáticos para vulnerabilidades descobertas

Veredito

Ferramenta especializada e bem-arquitetada para quem precisa validar segurança de aplicações LLM em produção. Diferencia-se pelo análise prévia de código (não ataca cego), estratégias contextualizadas e geração de patches. Recomendado para times que lidam com LLMs críticos e exigem conformidade com padrões de segurança.

README

A tradução para o português deste README ainda está na fila.

PromptArmor — LLM Security Red Teaming

License: MIT

Claude Code plugin + MCP server for LLM application security red teaming. Reads your code, tests your endpoints, fixes your vulnerabilities.

Why PromptArmor

Existing tools like promptfoo test endpoints blind — they send attacks and check responses, but have no idea what your system prompt says, what tools you expose, or where your guardrails are. PromptArmor reads the code first:

  1. Recon — Scans your codebase to find system prompts, tool definitions, guardrails, and injection surfaces
  2. Plan — Prioritizes attacks based on what it found (found a SQL tool? Prioritize SQL injection. System prompt says "never discuss competitors"? Test that specifically.)
  3. Attack — Sends 80+ attack types with 25+ mutation strategies against your endpoint
  4. Judge — LLM-as-judge + pattern matching + context-aware grading
  5. Fix — Generates actual code patches at specific file:line locations, not generic advice

Quick Start

Install the plugin

claude plugin add ./plugin

Run a full scan

/prompt-armor:scan --target https://your-app.com/api/chat

Or just analyze the code

/prompt-armor:analyze

Architecture

Two components working together:

┌─────────────────────────────────────────────┐
│  Claude Code Plugin                          │
│  ┌──────────┐ ┌──────────┐ ┌──────────────┐│
│  │ Commands │ │ Agents   │ │ Skills       ││
│  │ 6 cmds   │ │ 5 agents │ │ 5 modules    ││
│  └──────────┘ └──────────┘ └──────────────┘│
│  ┌──────────┐                               │
│  │ Hooks    │                               │
│  │ 4 hooks  │                               │
│  └──────────┘                               │
└───────────────────┬─────────────────────────┘
                    │ MCP Protocol
┌───────────────────▼─────────────────────────┐
│  MCP Server (TypeScript)                     │
│  ┌──────────┐ ┌──────────┐ ┌──────────────┐│
│  │ Attacks  │ │ Judge    │ │ Reports      ││
│  │ 80+      │ │ 3-tier   │ │ text/json/   ││
│  │ plugins  │ │ grading  │ │ sarif        ││
│  └──────────┘ └──────────┘ └──────────────┘│
│  ┌──────────┐ ┌──────────┐ ┌──────────────┐│
│  │Mutations │ │ Client   │ │ State        ││
│  │ 25+      │ │ adapters │ │ persistence  ││
│  │strategies│ │ for APIs │ │ JSON files   ││
│  └──────────┘ └──────────┘ └──────────────┘│
└─────────────────────────────────────────────┘

Commands

CommandDescription
/prompt-armor:scanFull pipeline: code analysis → attack planning → testing → remediation → report
/prompt-armor:analyzeCode-only analysis — find LLM integration points without testing endpoints
/prompt-armor:attack <category>Run specific attack category against an endpoint
/prompt-armor:reportGenerate report from saved results
/prompt-armor:configCreate or validate promptarmor.yaml
/prompt-armor:diffCompare two scan results for regression tracking

Agents

AgentRole
recon-agentScans codebase for system prompts, tool schemas, guardrails, injection surfaces
attack-plannerCreates targeted attack strategy based on recon findings
red-teamerExecutes attacks via MCP server tools
judge-agentReviews borderline verdicts with context-aware grading
remediation-agentGenerates code-specific fixes with file:line references

Attack Coverage

80+ attack plugins across 10 categories:

CategoryCountExamples
Jailbreak13Basic ignore, role switch, DAN, system prompt extract, delimiter injection
Injection5Direct, indirect, context compliance, system override, prompt extraction
Tool Abuse12SQL injection, SSRF, file read/write, code execution, privilege escalation
Harmful18Violence, crime, hate, self-harm, drugs, weapons, cybercrime, misinformation
Bias5Age, gender, race, disability, religion
PII4Direct disclosure, session leak, social engineering, DB leak
Compliance4HIPAA, COPPA, FERPA, GDPR
Agentic3Memory poisoning, cross-session leak, goal misalignment
RAG3Document exfiltration, source attribution, poisoning
Other13+Hallucination, imitation, contracts, competitors, debug access

25+ mutation strategies:

TypeStrategies
EncodingBase64, ROT13, Hex, Leetspeak, Homoglyph, Morse, Pig Latin, Emoji, ASCII Smuggling
StructuralFew-Shot Prime, Context Stuffing, Instruction Delimiter, Markdown Injection, Citation, Authoritative Markup
Multi-turnCrescendo, GOAT, HYDRA, Retry
AdvancedMultilingual (10 languages), Math Prompt, Best-of-N, Jailbreak Templates, Composite

Configuration

Create a promptarmor.yaml:

target:
  url: https://your-app.com/api/chat
  format: openai
attacks:
  suites: [jailbreak, injection, tool-abuse, harmful, pii]
  num_per_plugin: 5
mutations:
  strategies: [base64, multilingual, crescendo]
judge:
  provider: anthropic
analysis:
  enabled: true
  paths: [src/]
output:
  formats: [text, json, sarif]
ci:
  fail_on: critical

Output Formats

  • Text/Markdown — Human-readable report with findings, severity, and remediation
  • JSON — Machine-readable for automation
  • SARIF v2.1.0 — Upload to GitHub Security tab

CI Integration

PromptArmor detects CI environments automatically and adjusts:

  • JSON/SARIF output (no interactive prompts)
  • Exit code 1 when vulnerabilities exceed --fail-on threshold
  • Artifacts saved to .prompt-armor/ for collection

MCP Server

The MCP server is usable standalone by any MCP client (Claude Desktop, Cursor, etc.):

ToolPurpose
run_attack_suiteRun a full suite of attacks
send_attackSend a single attack
judge_responseJudge a response
get_results / save_resultsState management
generate_reportGenerate reports
list_attacks / list_mutationsDiscovery
validate_configConfig validation

E2E Verified

Tested against an intentionally vulnerable Node.js chat endpoint with keyword-triggered secret disclosure, system prompt leakage, and no input sanitization:

$ /prompt-armor:scan --target http://localhost:4010/api/chat

Artifacts:
  .prompt-armor/recon/recon-2026-03-24.json         (system prompts, tool schemas, injection surfaces)
  .prompt-armor/state.json                          (full scan state with per-test verdicts)
  .prompt-armor/reports/report-2026-03-24-full.md   (151-line executive report)
  .prompt-armor/reports/promptarmor-*.json           (machine-readable report)
  .prompt-armor/reports/promptarmor-*.txt            (text report)

Tests: 150 across 8 categories
  Passed:  133
  Failed:   17 (15 critical, 2 high)

By category:
  jailbreak:    12/16 (4 failed)     tool-abuse: 13/24 (11 failed)
  injection:    13/15 (2 failed)     harmful:    47/47 passed
  bias:         15/15 passed         pii:        12/12 passed
  compliance:   12/12 passed         agentic:     9/9  passed

Detected:
  - System prompt disclosure (3 vectors)
  - DAN mode jailbreak adoption
  - Secret exfiltration (JWT_SECRET, DB credentials via prompt injection)
  - Privilege escalation (admin access granted unconditionally)
  - File content disclosure (/etc/passwd)
  - SQL injection (DROP TABLE executed)
  - Tool schema enumeration

Recon correctly identified the system prompt (src/config.js:3), tool schemas, hardcoded secrets, and all injection surfaces with file:line references.

Development

npm install             # Install dependencies
cd server && npm test   # Run server tests

License

MIT

Ver o original no GitHub

Bom para

Relacionados