Avaliação de Prompts no AWS Bedrock
gamzeakkurt/Prompt-Evaluation-in-AWS-Bedrock
Framework de avaliação de LLMs com validação de código e grading por IA
Ver no GitHubFramework para avaliar outputs de LLMs usando AWS Bedrock com três abordagens: validação de código (JSON, Python, Regex), grading baseado em modelo (Claude 3.5 Haiku) e revisão humana. Inclui geração automática de datasets e scoring estruturado para tarefas determinísticas.
Instalação
npx convez-ai gamzeakkurt/Prompt-Evaluation-in-AWS-Bedrock1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Avaliar qualidade de saídas de modelos de IA em produção
- Validar código gerado automaticamente (Python, JSON, Regex)
- Implementar pipelines de QA para sistemas com LLMs
- Comparar performance de diferentes prompts e modelos
Veredito

Solução prática e bem estruturada para quem precisa validar outputs de LLMs em escala. Combina eficientemente validação determinística (rápida, barata) com grading semântico (preciso, custoso), permitindo estratégias híbridas. Ideal para pipelines de produção em AWS, mas requer familiaridade com Bedrock e Python.
README
A tradução para o português deste README ainda está na fila.
Prompt Evaluation in AWS Bedrock
This repository demonstrates practical approaches for evaluating AI-generated outputs using Amazon Bedrock. It focuses on lightweight, reproducible techniques to assess the quality of model responses for structured tasks such as generating Python code, JSON, and Regular Expressions.
🚀 Overview
Evaluating LLM outputs is critical for building reliable AI systems. This project implements three core grading strategies:
1. Code Graders
Programmatic validation using deterministic logic:
- JSON validation via parsing
- Python syntax validation via AST
- Regex validation via compilation
2. Model Graders
Use an LLM (Claude 3.5 Haiku via Bedrock) to:
- Analyze outputs
- Provide structured feedback
- Assign quality scores
3. Human Graders
Manual review for:
- Subjective quality assessment
- Edge cases
- Final validation (not automated in this repo, but supported conceptually)
🧠 Architecture
The workflow consists of:
-
Dataset Generation
- Automatically generate evaluation tasks using an LLM
-
Response Generation (Notebook)
- Prompts are executed and responses are collected
-
Evaluation
- Syntax-based grading (code graders)
- LLM-based grading (model graders)
🛠️ Setup
Prerequisites
- Python 3.9+
- AWS account with Bedrock access
boto3configured with credentials
Install Dependencies
pip install boto3
⚙️ Configuration
import boto3
client = boto3.client("bedrock-runtime", region_name="us-west-2")
# Claude 3.5 Haiku (fast + cost-effective for evaluation)
model_id = "us.anthropic.claude-3-5-haiku-20241022-v1:0"
💬 Chat Helper
Utility wrapper for interacting with Bedrock:
def chat(messages, system=None, temperature=1.0, stop_sequences=[]):
params = {
"modelId": model_id,
"messages": messages,
"inferenceConfig": {
"temperature": temperature,
"stopSequences": stop_sequences,
},
}
if system:
params["system"] = [{"text": system}]
response = client.converse(**params)
return response["output"]["message"]["content"][0]["text"]
📊 Dataset Generation
Generates structured evaluation tasks:
def generate_dataset():
...
Each dataset item includes:
task: Problem descriptionformat:python,json, orregexsolution_criteria: Evaluation guidelines
🤖 Model-Based Grading
Uses Claude to evaluate outputs:
def grade_by_model(test_case, output):
...
Output Format
{
"strengths": ["..."],
"weaknesses": ["..."],
"reasoning": "...",
"score": 8
}
🔍 Code-Based Grading
Fast, deterministic validation:
def grade_syntax(response, test_case):
...
Supported Validators
- ✅ JSON →
json.loads - ✅ Python →
ast.parse - ✅ Regex →
re.compile
Returns:
10→ Valid0→ Invalid
📓 Notebook
A Jupyter notebook (to be added) demonstrates:
- End-to-end evaluation workflow
- Prompt testing
- Scoring comparisons
📈 Evaluation Strategy
| Method | Speed | Cost | Accuracy | Use Case |
|---|---|---|---|---|
| Code Graders | ⚡ Fast | 💲 Low | ⚖️ Medium | Syntax validation |
| Model Graders | ⏱ Medium | 💲💲 | 🎯 High | Semantic evaluation |
| Human Review | 🐢 Slow | 💲💲💲 | 🧠 Highest | Final judgment |
🔮 Future Improvements
- Add human annotation interface
- Store evaluation results (e.g., DynamoDB / S3)
- Benchmark multiple models
- Add unit-test-based grading for Python tasks
📚 References
🤝 Contributing
Contributions are welcome! Feel free to open issues or submit pull requests to improve evaluation methods or extend support for additional formats.
⭐ Summary
This repo provides a simple but powerful framework for:
- Generating evaluation datasets
- Scoring LLM outputs
- Comparing grading strategies
It’s a practical starting point for anyone building LLM evaluation pipelines on AWS Bedrock.