convez.ai
PluginSegurançaAvançado
johnlorenthomas14

AI Security Portfolio

johnlorenthomas14/ai-security-portfolio

Detection engineering e compliance para segurança de modelos IA em produção

Ver no GitHub
InicialPluginScore · de 0 a 100

Conjunto de 9 projetos de segurança para workloads LLM, cobrindo detecção de prompt injection, auditoria de RAG, análise de gaps NIST AI RMF e geração de regras SIEM. Traduz 9+ anos de experiência em detection engineering federal/enterprise em um toolkit completo, com todos os projetos mapeados a frameworks reconhecidos (MITRE ATLAS, OWASP LLM Top 10, NIST 800-53).

Instalação

$npx convez-ai johnlorenthomas14/ai-security-portfolio

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
1
Forks
0
Acompanhando
0
Saúde
28%
Último commit
há 6 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Detectar e prevenir prompt injection em camada de entrada LLM
  • Auditar segurança de pipelines RAG antes de ingestão
  • Gerar regras SIEM correlacionadas para workloads IA
  • Validar conformidade NIST AI RMF com evidência contínua
  • Scan de supply chain de modelos (pickle, provenance, deps)
  • Red-teaming automatizado contra OWASP LLM Top 10

Veredito

Portfolio maduro e rigoroso, construído por especialista em detection engineering federal. Cada projeto é offline-first, testado em CI com floor de regressão, e mapeado explicitamente a frameworks de compliance reais (NIST, MITRE ATLAS, FedRAMP). Essencial para times de segurança que precisam defender LLMs em produção com evidence auditável.

README

A tradução para o português deste README ainda está na fila.

AI Security Portfolio

CI Python 3.11+ License: MIT

Nine projects translating 9+ years of federal/enterprise SIEM and Splunk detection-engineering experience into a complete AI Security Engineer toolkit. Each project is self-contained, runs offline by default, and maps to recognized control frameworks (MITRE ATLAS, OWASP LLM Top 10 (2025), NIST AI RMF 1.0, NIST 800-53 / FedRAMP).

Live site: johnlorenthomas14.github.io/ai-security-portfolio

See CLAUDE.md for conventions, threat-model template, and the detector output schema shared across the portfolio.

The lineup

#ProjectOne-lineStatus
1LLM Prompt Injection Detection PipelineHybrid heuristic + Claude-judge detector at the LLM input layer with optional NVIDIA NeMo Guardrails third layer. ATLAS-tagged verdicts forwarded to Splunk ES; F1 = 1.000 on shipped eval with CI regression floor.Working MVP
2AI Red-Team Automation FrameworkProbes any LLM target against OWASP LLM Top 10; three-layer scorer reuses Project 1's detector. JSON findings + AI-RMF Markdown evidence.Working MVP
3RAG Pipeline Security AuditorStatic corpus scanner — indirect-injection, PII, credentials, sensitivity markers caught at ingestion before they reach a production LLM.Working MVP
4SIEM Correlation Rule Generator for AI WorkloadsOne canonical YAML rule → four SIEM-native outputs (Splunk ES correlation searches, Sigma rules, Cortex XSIAM XQL, NVIDIA Morpheus pipeline configs). Auto-generated AI-RMF coverage map.Working MVP
5LLM Output Safety & Compliance MonitorRuntime output guardrail with PII / secrets / classification / policy filters plus optional NVIDIA NeMo Guardrails fifth filter. Hash-chained, tamper-evident audit log — the FedRAMP-flavored integrity-evidence pattern.Working MVP
6AI Model Supply Chain Risk ScannerPickle-opcode disassembly without execution + provenance + risky-deps scan of model artifacts; CycloneDX 1.5 SBOM fragment output.Working MVP
7Adaptive Threat Intelligence SummarizerSTIX 2.1 → cited analyst briefs + Splunk notables. Citation verifier rejects hallucinated references — every claim Claude writes is bound to a real bundle object ID.Working MVP
8NIST AI RMF Compliance Gap AnalyzerReads outputs from Projects 1–7, maps each finding to its AI RMF subcategory, produces continuous-monitoring evidence + gap report. The Qmulos Q-Compliance / Q-Audit pattern, applied to AI governance.Working MVP
9Autonomous SOC Analyst AgentAgentic tool-use loop with six analyst tools, defended at the input layer by Project 1's detector. 8-case eval suite at 100% pass rate, CI-gated at 75% floor.Working MVP

Cross-cutting layer: nemo_guardrails/ — an NVIDIA NeMo Guardrails app (Colang flows + Python actions) wired as an opt-in third detection layer in Project 1 and an opt-in fifth output filter in Project 5. Demonstrates programmable AI-safety policy as a peer to the imperative-detector pattern.

The closed-loop pipeline

Nine projects across three deployment phases, all sharing an event vocabulary, a taxonomy (OWASP LLM Top 10 / MITRE ATLAS / NIST AI RMF), and a Splunk ingestion story:

                 PRE-DEPLOYMENT                        RUNTIME                       GOVERNANCE / OPERATIONS
            ┌────────────────────────┐         ┌────────────────────────┐        ┌────────────────────────┐
            │ 06 — Model Supply      │         │ 01 — Prompt Injection  │        │ 04 — SIEM Correlation  │
            │      Chain Scanner     │         │      Detection         │        │      Rule Generator    │
            │   (pickle / SBOM)      │         │   (input layer)        │        │   (Splunk + Sigma +    │
            │                        │         │                        │        │    XSIAM + Morpheus)   │
            ├────────────────────────┤         ├────────────────────────┤        ├────────────────────────┤
            │ 03 — RAG Pipeline      │         │ 05 — Output Safety &   │        │ 07 — Threat-Intel      │
            │      Security Auditor  │   ───►  │      Compliance        │  ───►  │      Summarizer        │
            │   (static corpus       │         │      Monitor           │        │   (STIX → notables)    │
            │    scanning)           │         │   (output layer +      │        │                        │
            │                        │         │    hash-chained log)   │        ├────────────────────────┤
            └────────────────────────┘         ├────────────────────────┤        │ 08 — NIST AI RMF       │
                                               │ 02 — AI Red-Team       │        │      Gap Analyzer      │
                                               │      Framework         │        │   (continuous-evidence │
                                               │   (adversarial         │        │    capstone)           │
                                               │    validation)         │        │                        │
                                               ├────────────────────────┤        ├────────────────────────┤
                                               │ 09 — Autonomous SOC    │        │                        │
                                               │      Analyst Agent     │ ◄──────│  (consumes notables    │
                                               │   (defended by P1)     │        │   from P4 + P7)        │
                                               └────────────────────────┘        └────────────────────────┘

The compounding interview line, in one paragraph:

"Project 6 audits the model artifact pre-deployment. Project 3 audits the corpus pre-retrieval. Project 1 detects injection at runtime. Project 5 guards the output layer with hash-chained tamper-evident audit logs. Project 2 generates the adversarial corpus that keeps Project 1's eval set honest. Project 7 produces citation-verified threat-intel briefs from STIX bundles. Project 4 turns every project's findings into deployable Splunk ES, Sigma, Cortex XSIAM, and NVIDIA Morpheus correlation content. Project 8 reads the whole pipeline's output and produces NIST AI RMF continuous-monitoring evidence. Project 9 is an autonomous SOC analyst agent that defends its own input layer using Project 1 — closed-loop AI-security as defense-in-depth."

Engineering write-ups

Three projects ship long-form engineering write-ups (each ~3,000 words):

Quick start

# Clone
git clone https://github.com/johnlorenthomas14/ai-security-portfolio
cd ai-security-portfolio

# Project 1 — prompt-injection detector
cd 01-prompt-injection-pipeline && pip install -r requirements.txt && ./demo.sh

# Project 2 — red-team probes against the in-process vulnerable demo target
cd ../02-ai-red-team-framework && pip install -r requirements.txt && python3 runner.py --target demo

# Project 3 — audit the shipped polluted demo corpus
cd ../03-rag-security-auditor && pip install -r requirements.txt && python3 auditor.py --corpus corpus/demo

# Project 4 — generate detection content for four platforms from one canonical rule library
cd ../04-siem-correlation-generator && pip install -r requirements.txt && python3 generator.py

# Project 5 — replay LLM output samples through the compliance monitor
cd ../05-llm-output-compliance-monitor && pip install -r requirements.txt && python3 monitor.py --corpus samples/bad_outputs.jsonl --audit-log out/audit.jsonl

# Project 6 — scan a model directory for pickle / provenance / dependency risk
cd ../06-model-supply-chain-scanner && pip install -r requirements.txt && python3 scanner.py --model samples/demo_model

# Project 7 — generate cited threat-intel briefs offline
cd ../07-threat-intel-summarizer && pip install -r requirements.txt && python3 summarizer.py --offline

# Project 8 — capstone: read everything Projects 1–7 produced and emit a NIST AI RMF coverage report
cd ../08-nist-ai-rmf-gap-analyzer && pip install -r requirements.txt && python3 analyzer.py

# Project 9 — autonomous SOC analyst agent over notable events
cd ../09-autonomous-soc-analyst && pip install -r requirements.txt && python3 analyst.py

Every project ships its own demo.sh for asciinema recording / screen capture, its own pytest suite that runs offline, and CIM-shaped JSON output ready for Splunk HEC ingestion.

Why this shape

Each project pairs an AI-security capability with a SOC-team output: a detection rule, a notable event, an analyst summary, a compliance artifact. That's deliberate — the portfolio reads as detection engineering that happens to target AI workloads, not as ML demos that mention security.

The output discipline is consistent across all nine projects:

  • Threat model up front in every README — attacker capability, target asset, in-scope, out-of-scope. Scope honesty is the price of being trusted as a control.
  • Measurable detector with a regression-tested eval where applicable. CI enforces a numeric floor; performance regressions break the build.
  • CIM-aligned JSON as the primary output. Drop-in Splunk HEC ingestion, no parsing required at the indexer.
  • AI-RMF-aligned Markdown as the secondary output. The same detection content doubles as continuous-monitoring evidence for the Measure and Manage functions of NIST AI RMF.
  • Cortex XSIAM mapping documented in the cross-platform projects. Detection content authored once, portable across SIEM stacks.
  • Offline-by-default operation. Every project's CI run is deterministic, reproducible, and requires no API key — the regression guard is real, not aspirational.

CI

Eighteen jobs per push (9 projects × Python 3.11 / 3.12). Every project has at least:

  • A pytest -q step that runs the full test suite offline.
  • An end-to-end demo run that exercises the project's CLI and verifies the expected output artifacts are produced.

For the projects that have measurable detectors (Projects 1, 9), CI also enforces a regression floor — the build breaks if accuracy or pass-rate drops below threshold.

Author

John Thomas — Principal Splunk Consultant, 9+ years federal SIEM (IRS, federal health, federal oversight) and Fortune-500 enterprise detection engineering. Currently extending the same NIST 800-53 / FedRAMP detection-engineering discipline into AI / LLM security.

Ver o original no GitHub

Bom para

Relacionados