convez.ai
ComandoDevOps & InfraestruturaIntermediário
Code-and-Sorts

PromptDrifter

Code-and-Sorts/PromptDrifter

Detecta mudanças em respostas de LLM e falha o build automaticamente

Ver no GitHub
BomPluginScore · de 0 a 100

Guardrail de CI em um comando que detecta mudanças indesejadas nas respostas de modelos de linguagem e falha o build automaticamente. Monitora prompt drift monitorando alterações em saídas de LLM entre versões, prevenindo regressões silenciosas em pipelines que dependem de respostas consistentes.

Instalação

$npx convez-ai Code-and-Sorts/PromptDrifter

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
9
Forks
0
Acompanhando
1
Saúde
75%
Último commit
há 1 dia
Versão
—
Issues abertas
5
Contribuidores
3

Casos de uso

  • Validar consistência de saídas de LLM em CI/CD
  • Prevenir regressões causadas por atualizações de modelos
  • Monitorar alterações comportamentais de prompts em produção

Veredito

Ferramenta prática e necessária para times que usam LLMs em produção, abordando um problema real de drift de comportamento. Bem estruturada com CLI simples, mas ainda em desenvolvimento ativo com possibilidade de breaking changes.

README

A tradução para o português deste README ainda está na fila.

PromptDrifter is a one-command CI guardrail, open source platform for catching prompt drift and fails if your LLM answers change.

[!IMPORTANT] Development Notice: This project is under active development. Breaking changes may occur between versions. Please check the changelog and release notes before updating.

Quick-Start

Basic Installation

pip install promptdrifter

With Text Similarity Support

pip install 'promptdrifter[similarity]'

Development Installation

pip install 'promptdrifter[dev,similarity]'

To start using PromptDrifter:

  1. Initialize a sample configuration:
    promptdrifter init
    

This will create a promptdrifter.yaml file in your current directory.

  1. Edit promptdrifter.yaml to define your prompts and expected outcomes.

  2. Run your tests:

    promptdrifter run promptdrifter.yaml
    

Sample Run

Why PromptDrifter?

The landscape of Large Language Models (LLMs) is one of rapid evolution. While exciting, this constant change introduces a critical challenge for applications relying on them: prompt drift.

Over time, updates to LLM versions, or even subtle shifts in their training data or internal architecture, can cause their responses to identical prompts to change. These changes can range from minor formatting differences to significant alterations in content or structure, potentially breaking downstream processes causing issues with the integrity of your application.


Undetected prompt drift can lead to:


PromptDrifter tackles these challenges head-on by providing:

[!NOTE] By making prompt-response testing a straightforward and automated part of your workflow, PromptDrifter helps you harness the power of LLMs while mitigating the risks associated with their dynamic nature.

Demo

Running with cache

Running without cache

Running and experiencing a failure

Supported LLM Adapters

PromptDrifter is designed to be extensible to various Large Language Models through its adapter system. Here's a current list of supported and planned adapters:

Provider / Model FamilyAdapter StatusDetails / Model ExamplesLinked Issue
GPT (OpenAI)✅ Availablegpt-3.5-turbo, gpt-4, gpt-4o, etc.N/A
Azure OpenAI✅ Availablegpt-3.5-turbo, gpt-4, gpt-4o, etc.N/A
Ollama✅ Availablellama3, mistral, gemma, etc.N/A
Claude (Anthropic)✅ Availableclaude-3-7-sonnet, claude-3-5-sonnet, claude-3-opusN/A
Gemini (Google)✅ Availablegemini-2.5-pro, gemini-2.5-flash, gemini-2.0-flash-thinkingN/A
Qwen (Alibaba)✅ Availableqwen3-30b-a3b, qwq-32bN/A
Grok (xAI)✅ Availablegrok-3, grok-2, etc.N/A
DeepSeek✅ Availabledeepseek-r1, deepseek-v3-0324, etc.N/A
Mistral✅ Availablemistral-small-24b-instruct-2501, mistral-small-3.1-24b-instruct-2503N/A
Llama (Meta)📋 To Dollama-4-maverick, llama-4-scout, etc.#11

If there's a model or provider you'd like to see supported, please open a feature request or consider contributing an adapter!

Supported Drift Tests

NameConfig keyDescriptionInstallation
Exact Matchexpect_exactOutput should be an exact match✅ Core
Regexexpect_regexOutput should match regex pattern✅ Core
Substringexpect_substringOutput should contain the substring✅ Core
Substring Case Insensitiveexpect_substring_case_insensitiveCase insensitive substring match✅ Core
Text Similaritytext_similaritySemantic similarity using sentence transformerspip install 'promptdrifter[similarity]'

GitHub Action

Automate your prompt drift detection by integrating PromptDrifter directly into your GitHub workflows!

We provide a reusable GitHub Action that makes it easy to run your PromptDrifter tests on every push or pull request.

Find the PromptDrifter GitHub Action here: PromptDrifter on GitHub Marketplace

Basic Usage

Add this to your workflow file (e.g., .github/workflows/prompt-drift.yml):

name: Prompt Drift Detection

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  prompt-drift-test:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v4

    - name: Run PromptDrifter
      uses: Code-and-Sorts/PromptDrifter-action@v0.0.2
      with:
        files: 'tests/promptdrifter.yaml'
        openai-api-key: ${{ secrets.OPENAI_API_KEY }}

Features

This action allows you to:

  • Install a specific version of PromptDrifter or use the latest
  • Specify your test files and configurations
  • Support multiple LLM providers (OpenAI, Claude, Gemini, etc.)
  • Secure API key handling via GitHub secrets

By using the action, you can ensure that any changes to your LLM's responses that violate your defined tests will automatically flag your CI builds, preventing unexpected issues from reaching production.

Docs

Our documentation is built with Docusaurus and deployed to GitHub Pages.

You can view the full documentation at: https://code-and-sorts.github.io/PromptDrifter/

The documentation includes:

  • Getting Started guide
  • Configuration options
  • API reference
  • Examples and tutorials

If you want to contribute to the documentation, see the docs/README.md file for instructions.

Contributing

Follow the contributing guide.

Code of Conduct

Please make sure you read the Code of Conduct guide.

Changelog

Ver o original no GitHub

Bom para

Relacionados