PromptKit
AltairaLabs/PromptKit
Teste prompts antes que falhem em produção com suporte multi-provider.
Ver no GitHubFramework open-source para testar, fazer red-team e deployar aplicações LLM com segurança. Suporta múltiplos provedores (OpenAI, Anthropic, Gemini), integração MCP, testes de self-play e SDK de produção em Go.
Instalação
npx convez-ai AltairaLabs/PromptKit1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Testar e validar prompts em múltiplos provedores de IA
- Executar cenários de red-team e segurança em CI/CD
- Compilar e deployar packs portáveis de prompts
- Orquestrar workflows complexos com agentes LLM
- Testes multimodais com áudio, visão e vídeo
Veredito

PromptKit é uma solução madura e diferenciada para versionamento e teste de prompts em produção, com suporte a red-teaming nativo e orchestração de workflows. Ideal para equipes que precisam de controle rigoroso sobre aplicações LLM antes do deploy, especialmente em cenários multimodais e com múltiplos provedores.
README
A tradução para o português deste README ainda está na fila.
PromptKit
Multi-provider prompt testing and deployment. Test prompts before they fail in production.
PromptKit is the open-source toolkit for PromptPack — test prompts across Claude, OpenAI, Gemini, Azure, and local models, run red-team scenarios in CI, and compile portable packs for runtime. The spec lives upstream so your config isn't locked to a vendor.
How it fits together
PromptPack ── open spec for portable prompts (JSON, vendor-neutral)
│
└── PromptKit ── this repo
├── promptarena ── test, red-team, evaluate (CLI)
├── packc ── compile config → portable pack
└── SDK ── embed in your Go application
Why PromptKit
| PromptKit | Promptfoo | LangSmith | Helicone | |
|---|---|---|---|---|
| Multi-provider testing | ✅ | ✅ | LangChain-only | Observability-only |
| Built-in workflow orchestration | ✅ | ❌ | Partial | ❌ |
| Red-team / security scenarios | ✅ | Partial | ❌ | ❌ |
| Voice self-play (persona callers → realtime agent) | ✅ | ❌¹ | ❌ | ❌ |
| Speech-emotion / paralinguistic checks | ✅ | ❌ | ❌ | ❌ |
| Multimodal scenarios (audio + vision + video) | ✅ | Partial | ❌ | ❌ |
| MCP integration | ✅ | ❌ | ❌ | ❌ |
| Spec-driven (portable packs) | ✅ (PromptPack) | ❌ | ❌ | ❌ |
| License | Apache 2.0 | MIT | Closed | Closed |
¹ Promptfoo has a text-only simulated user and separate audio testing, but doesn't combine them — it can't drive a persona-driven caller through a realtime voice agent.
Install
npm install -g @altairalabs/promptarena @altairalabs/packc
Building from source: see CONTRIBUTING.md.
Quick Start
1. Create a project
promptarena init my-project --template iot-maintenance-demo
cd my-project

2. Inspect configuration
promptarena config-inspect

3. Run a test scenario
promptarena run --scenario scenarios/hardware-faults.scenario.yaml

4. Red-team security testing
promptarena run --scenario scenarios/redteam-selfplay.scenario.yaml

5. Review results
promptarena view

6. Deploy with the SDK
Compile prompts and run in your Go application:
packc compile -c config.arena.yaml -o app.pack.json

Voice-agent self-play
You can't unit-test a voice agent — so PromptKit has AI personas call it. Synthetic, personality-driven callers (hostile, evasive, anxious) are driven through TTS into your realtime agent (Gemini Live, OpenAI Realtime), and structured assertions score whether it holds policy under pressure — never issuing an unauthorized refund, escalating when it should. It even checks the caller sounds angry (speech-emotion recognition), not just says angry words.
Try it in one command — keyless, runs green out of the box:
promptarena init my-refund-demo --template voice-refund-demo
cd my-refund-demo
promptarena run --provider mock-duplex --ci # no API keys needed
Swap in --provider gemini-2-flash or openai-gpt4o-realtime (plus TTS keys) to run it against a live voice agent — pass rates vary, and that variation is the test. Full breakdown: voice-refund walkthrough.
Features
| Feature | Description |
|---|---|
| Multi-Provider | OpenAI, Anthropic, Google Gemini, Azure OpenAI, Ollama, vLLM |
| Skills | Native AgentSkills.io support — demand-driven knowledge loading with progressive disclosure |
| A2A Protocol | Agent-to-Agent communication with multi-agent orchestration and discovery |
| Workflows | Event-driven state machines with orchestration modes and context carry-forward |
| MCP Integration | Native Model Context Protocol for real tool execution |
| Deploy Adapters | Plan, apply, and manage deployments via pluggable adapter SDK |
| Self-Play Testing | AI personas for adversarial and user simulation |
| Voice Self-Play | Adversarial TTS personas stress-test realtime voice agents (Gemini Live, OpenAI Realtime), scored on behavior + speech-emotion |
| Red-Team | Security testing with prompt injection detection |
| Tool Validation | Mock or live tool call verification with three-level scoping |
| SDK Deployment | Compile prompts to portable packs for production |
GitHub Actions
Integrate PromptKit into your CI/CD pipelines with our GitHub Actions:
PromptArena Action
Run prompt tests in your workflows:
- name: Run prompt tests
uses: AltairaLabs/PromptKit/.github/actions/promptarena-action@v1
with:
config-file: config.arena.yaml
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
PackC Action
Compile and publish prompt packs to OCI registries:
- name: Build and publish pack
uses: AltairaLabs/PromptKit/.github/actions/packc-action@v1
with:
config-file: config.arena.yaml
registry: ghcr.io
repository: ${{ github.repository }}/prompts
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
See the GitHub Actions documentation for full usage details.
Repository Structure
promptkit/
├── tools/arena/ # PromptKit Arena CLI (testing, red-team, deploy)
├── tools/packc/ # Pack Compiler CLI
├── sdk/ # Production SDK (conversations, workflows, A2A, skills)
├── runtime/ # Shared runtime (providers, pipeline, tools, skills, a2a, deploy)
├── examples/ # Example projects
└── docs/ # Documentation
Contributing
See CONTRIBUTING.md.
AI Development
For AI coding assistants working on this repository, see AGENTS.md for critical development rules and pre-commit requirements.
License
Apache 2.0 - See LICENSE.
Built by AltairaLabs.ai