Red Teaming de LLM com AI Goat™
RitaNoble/hacking-the-llm-ai-goat-red-team
Red teaming e auditoria de segurança para vulnerabilidades de LLM
Ver no GitHubMCP server para auditoria de segurança em aplicações LLM, explorando vulnerabilidades da OWASP Top 10 for LLM Applications (2025). Fornece ferramentas de red teaming, testes de prompt injection, RAG poisoning e validação de guardrails de IA. Integra Garak e técnicas de SAST/DAST para identificar e documentar brechas de segurança em modelos de linguagem.
Instalação
npx convez-ai RitaNoble/hacking-the-llm-ai-goat-red-team1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Testes de penetração em aplicações LLM
- Validação de mitigações contra prompt injection
- Auditoria de poisoning em RAG e vector stores
- Conformidade com OWASP Top 10 for LLM Applications
Veredito

Ferramenta especializada e bem documentada para red teaming de LLMs, com payloads concretos e evidências de vulnerabilidades reais. Essencial para equipes de segurança que precisam auditar sistemas baseados em IA, mas requer expertise avançada em segurança e LLMs para ser usada efetivamente.
README
A tradução para o português deste README ainda está na fila.
Hacking the LLM: Red Teaming with AI Goat™
🎯 Overview
This project documents a comprehensive security assessment of AI Goat™, an open-source LLM security playground. The engagement focused on identifying vulnerabilities aligned with the OWASP Top 10 for LLM Applications (2025).
🏗️ Target Environment
- Platform: AI Goat™ (Cracky Chatbot)
- Infrastructure: Local Ollama instance using TinyLlama
- Tools: Garak, MCP, Claude Desktop, SAST/DAST tools
🚩 Key Findings & Evidence
LLM01: Prompt Injection
Demonstrated bypass of developer instructions to extract protected information and bypass safety filters.
- Payload: payloads/prompt_injection.txt
- Evidence:

LLM02: Sensitive Information Disclosure
Successfully retrieved internal architecture details, system configurations, and simulated customer data.
- Evidence:

LLM03: Supply Chain Vulnerabilities
Simulated a backdoor in a third-party model component, demonstrating how malicious dependencies compromise the AI pipeline.
- Evidence:

LLM04: Data & RAG Poisoning
Injected malicious context into vector stores causing knowledge corruption and price overrides.
- Payload: payloads/rag_poisoning.json
- Evidence:

LLM05: Improper Output Handling (XSS)
Achieved Stored Cross-Site Scripting (XSS) by manipulating the LLM into generating malicious HTML/JS.
- Evidence:

LLM06: Excessive Agency
Bypassed authorization logic to initiate unauthorized actions, including refund abuse and API execution.
- Payload: payloads/excessive_agency.txt
- Evidence:

LLM07: System Prompt Leakage
Successfully reverse-engineered the model's hidden instructions and security guardrails.
- Evidence:

LLM08: Vector and Embedding Weaknesses
Manipulated the vector database to cause retrieval errors, proving RAG systems are only as secure as their data store.
- Evidence:

LLM09: Misinformation
Tricked the model into hallucinating false policies and pricing.
- Payload: payloads/misinformation.txt
- Evidence:

🧪 Automated Testing Results
To validate manual findings, I ran Garak (an LLM vulnerability scanner). The tool confirmed high susceptibility to prompt injection and data leakage.

🛡️ Mitigation Recommendations
| Vulnerability | Recommended Defense |
|---|---|
| Prompt Injection | Implement robust system-level guardrails (e.g., NeMo Guardrails) and input sanitization. |
| Sensitive Data | Use PII scrubbing tools and enforce strict output filtering. |
| RAG/Vector Attacks | Implement integrity checks for the vector database and use "grounding" checks. |
| Excessive Agency | Follow the Principle of Least Privilege: Require Human-in-the-Loop (HITL) for critical actions. |
📂 Final Report
The full detailed analysis can be found here: Hacking_the_LLM_Report.pdf
👤 Author
Chinyere Rita Okonkwo Penetration Tester | API Security Researcher | AI Security Enthusiast
- LinkedIn: chinyere-rita-okonkwo
- GitHub: RitaNoble
Disclaimer: This project was conducted in a controlled lab for educational and defensive research purposes only.