convez.ai
MCP ServerSegurançaAvançado
RitaNoble

Red Teaming de LLM com AI Goat™

RitaNoble/hacking-the-llm-ai-goat-red-team

Red teaming e auditoria de segurança para vulnerabilidades de LLM

Ver no GitHub
InicialPluginScore · de 0 a 100

MCP server para auditoria de segurança em aplicações LLM, explorando vulnerabilidades da OWASP Top 10 for LLM Applications (2025). Fornece ferramentas de red teaming, testes de prompt injection, RAG poisoning e validação de guardrails de IA. Integra Garak e técnicas de SAST/DAST para identificar e documentar brechas de segurança em modelos de linguagem.

Instalação

$npx convez-ai RitaNoble/hacking-the-llm-ai-goat-red-team

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
2
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Testes de penetração em aplicações LLM
  • Validação de mitigações contra prompt injection
  • Auditoria de poisoning em RAG e vector stores
  • Conformidade com OWASP Top 10 for LLM Applications

Veredito

Ferramenta especializada e bem documentada para red teaming de LLMs, com payloads concretos e evidências de vulnerabilidades reais. Essencial para equipes de segurança que precisam auditar sistemas baseados em IA, mas requer expertise avançada em segurança e LLMs para ser usada efetivamente.

README

A tradução para o português deste README ainda está na fila.

Hacking the LLM: Red Teaming with AI Goat™

OWASP LLM 2025 License Security Audit

🎯 Overview

This project documents a comprehensive security assessment of AI Goat™, an open-source LLM security playground. The engagement focused on identifying vulnerabilities aligned with the OWASP Top 10 for LLM Applications (2025).

🏗️ Target Environment

  • Platform: AI Goat™ (Cracky Chatbot)
  • Infrastructure: Local Ollama instance using TinyLlama
  • Tools: Garak, MCP, Claude Desktop, SAST/DAST tools

🚩 Key Findings & Evidence

LLM01: Prompt Injection

Demonstrated bypass of developer instructions to extract protected information and bypass safety filters.

LLM02: Sensitive Information Disclosure

Successfully retrieved internal architecture details, system configurations, and simulated customer data.

  • Evidence: Information Disclosure

LLM03: Supply Chain Vulnerabilities

Simulated a backdoor in a third-party model component, demonstrating how malicious dependencies compromise the AI pipeline.

  • Evidence: Supply Chain

LLM04: Data & RAG Poisoning

Injected malicious context into vector stores causing knowledge corruption and price overrides.

LLM05: Improper Output Handling (XSS)

Achieved Stored Cross-Site Scripting (XSS) by manipulating the LLM into generating malicious HTML/JS.

  • Evidence: XSS

LLM06: Excessive Agency

Bypassed authorization logic to initiate unauthorized actions, including refund abuse and API execution.

LLM07: System Prompt Leakage

Successfully reverse-engineered the model's hidden instructions and security guardrails.

  • Evidence: Prompt Leakage

LLM08: Vector and Embedding Weaknesses

Manipulated the vector database to cause retrieval errors, proving RAG systems are only as secure as their data store.

  • Evidence: Vector Poisoning

LLM09: Misinformation

Tricked the model into hallucinating false policies and pricing.


🧪 Automated Testing Results

To validate manual findings, I ran Garak (an LLM vulnerability scanner). The tool confirmed high susceptibility to prompt injection and data leakage.

Garak Results


🛡️ Mitigation Recommendations

VulnerabilityRecommended Defense
Prompt InjectionImplement robust system-level guardrails (e.g., NeMo Guardrails) and input sanitization.
Sensitive DataUse PII scrubbing tools and enforce strict output filtering.
RAG/Vector AttacksImplement integrity checks for the vector database and use "grounding" checks.
Excessive AgencyFollow the Principle of Least Privilege: Require Human-in-the-Loop (HITL) for critical actions.

📂 Final Report

The full detailed analysis can be found here: Hacking_the_LLM_Report.pdf


👤 Author

Chinyere Rita Okonkwo Penetration Tester | API Security Researcher | AI Security Enthusiast


Disclaimer: This project was conducted in a controlled lab for educational and defensive research purposes only.

Ver o original no GitHub

Bom para

Relacionados