convez.ai
MCP ServerDevOps & InfraestruturaAvançado
AltairaLabs

PromptKit

AltairaLabs/PromptKit

Teste prompts antes que falhem em produção com suporte multi-provider.

Ver no GitHub
BomPluginScore · de 0 a 100

Framework open-source para testar, fazer red-team e deployar aplicações LLM com segurança. Suporta múltiplos provedores (OpenAI, Anthropic, Gemini), integração MCP, testes de self-play e SDK de produção em Go.

Instalação

$npx convez-ai AltairaLabs/PromptKit

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
12
Forks
3
Acompanhando
0
Saúde
100%
Último commit
há 1 dia
Versão
v2.21.0
Issues abertas
40
Contribuidores
3

Casos de uso

  • Testar e validar prompts em múltiplos provedores de IA
  • Executar cenários de red-team e segurança em CI/CD
  • Compilar e deployar packs portáveis de prompts
  • Orquestrar workflows complexos com agentes LLM
  • Testes multimodais com áudio, visão e vídeo

Veredito

PromptKit é uma solução madura e diferenciada para versionamento e teste de prompts em produção, com suporte a red-teaming nativo e orchestração de workflows. Ideal para equipes que precisam de controle rigoroso sobre aplicações LLM antes do deploy, especialmente em cenários multimodais e com múltiplos provedores.

README

A tradução para o português deste README ainda está na fila.

PromptKit

CI Provider Matrix Docs Quality Gate Status Coverage Go Report Card Go Reference License: Apache 2.0

Multi-provider prompt testing and deployment. Test prompts before they fail in production.

PromptKit is the open-source toolkit for PromptPack — test prompts across Claude, OpenAI, Gemini, Azure, and local models, run red-team scenarios in CI, and compile portable packs for runtime. The spec lives upstream so your config isn't locked to a vendor.

How it fits together

PromptPack  ── open spec for portable prompts (JSON, vendor-neutral)
    │
    └── PromptKit  ── this repo
         ├── promptarena  ── test, red-team, evaluate (CLI)
         ├── packc        ── compile config → portable pack
         └── SDK          ── embed in your Go application

Why PromptKit

PromptKitPromptfooLangSmithHelicone
Multi-provider testing✅✅LangChain-onlyObservability-only
Built-in workflow orchestration✅❌Partial❌
Red-team / security scenarios✅Partial❌❌
Voice self-play (persona callers → realtime agent)✅❌¹❌❌
Speech-emotion / paralinguistic checks✅❌❌❌
Multimodal scenarios (audio + vision + video)✅Partial❌❌
MCP integration✅❌❌❌
Spec-driven (portable packs)✅ (PromptPack)❌❌❌
LicenseApache 2.0MITClosedClosed

¹ Promptfoo has a text-only simulated user and separate audio testing, but doesn't combine them — it can't drive a persona-driven caller through a realtime voice agent.

Install

npm install -g @altairalabs/promptarena @altairalabs/packc

Building from source: see CONTRIBUTING.md.

Quick Start

1. Create a project

promptarena init my-project --template iot-maintenance-demo
cd my-project

init project

2. Inspect configuration

promptarena config-inspect

config overview

3. Run a test scenario

promptarena run --scenario scenarios/hardware-faults.scenario.yaml

run scenario

4. Red-team security testing

promptarena run --scenario scenarios/redteam-selfplay.scenario.yaml

redteam test

5. Review results

promptarena view

view conversation

6. Deploy with the SDK

Compile prompts and run in your Go application:

packc compile -c config.arena.yaml -o app.pack.json

sdk demo

Voice-agent self-play

You can't unit-test a voice agent — so PromptKit has AI personas call it. Synthetic, personality-driven callers (hostile, evasive, anxious) are driven through TTS into your realtime agent (Gemini Live, OpenAI Realtime), and structured assertions score whether it holds policy under pressure — never issuing an unauthorized refund, escalating when it should. It even checks the caller sounds angry (speech-emotion recognition), not just says angry words.

Try it in one command — keyless, runs green out of the box:

promptarena init my-refund-demo --template voice-refund-demo
cd my-refund-demo
promptarena run --provider mock-duplex --ci   # no API keys needed

Swap in --provider gemini-2-flash or openai-gpt4o-realtime (plus TTS keys) to run it against a live voice agent — pass rates vary, and that variation is the test. Full breakdown: voice-refund walkthrough.

Features

FeatureDescription
Multi-ProviderOpenAI, Anthropic, Google Gemini, Azure OpenAI, Ollama, vLLM
SkillsNative AgentSkills.io support — demand-driven knowledge loading with progressive disclosure
A2A ProtocolAgent-to-Agent communication with multi-agent orchestration and discovery
WorkflowsEvent-driven state machines with orchestration modes and context carry-forward
MCP IntegrationNative Model Context Protocol for real tool execution
Deploy AdaptersPlan, apply, and manage deployments via pluggable adapter SDK
Self-Play TestingAI personas for adversarial and user simulation
Voice Self-PlayAdversarial TTS personas stress-test realtime voice agents (Gemini Live, OpenAI Realtime), scored on behavior + speech-emotion
Red-TeamSecurity testing with prompt injection detection
Tool ValidationMock or live tool call verification with three-level scoping
SDK DeploymentCompile prompts to portable packs for production

GitHub Actions

Integrate PromptKit into your CI/CD pipelines with our GitHub Actions:

PromptArena Action

Run prompt tests in your workflows:

- name: Run prompt tests
  uses: AltairaLabs/PromptKit/.github/actions/promptarena-action@v1
  with:
    config-file: config.arena.yaml
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

PackC Action

Compile and publish prompt packs to OCI registries:

- name: Build and publish pack
  uses: AltairaLabs/PromptKit/.github/actions/packc-action@v1
  with:
    config-file: config.arena.yaml
    registry: ghcr.io
    repository: ${{ github.repository }}/prompts
    username: ${{ github.actor }}
    password: ${{ secrets.GITHUB_TOKEN }}

See the GitHub Actions documentation for full usage details.

Repository Structure

promptkit/
├── tools/arena/     # PromptKit Arena CLI (testing, red-team, deploy)
├── tools/packc/     # Pack Compiler CLI
├── sdk/             # Production SDK (conversations, workflows, A2A, skills)
├── runtime/         # Shared runtime (providers, pipeline, tools, skills, a2a, deploy)
├── examples/        # Example projects
└── docs/            # Documentation

Contributing

See CONTRIBUTING.md.

AI Development

For AI coding assistants working on this repository, see AGENTS.md for critical development rules and pre-commit requirements.

License

Apache 2.0 - See LICENSE.


Built by AltairaLabs.ai

Ver o original no GitHub

Bom para

Relacionados