convez.ai
AgenteDados & AnalyticsIntermediário
Genevieve-Nartey

Autonomous Data Quality Agent

Genevieve-Nartey/autonomous-data-quality-agent

Auditoria autônoma de qualidade de dados com scoring e plano de remediação

Ver no GitHub
InicialPluginScore · de 0 a 100

Agent agentico que audita autonomamente datasets tabulares, detectando problemas de qualidade de dados (valores faltantes, duplicatas, outliers, violações de formato) e gerando relatório de remediação priorizado com score 0-100. Utiliza Claude Sonnet 4.5 para executar 7 verificações especializadas sem configuração manual.

Instalação

$npx convez-ai Genevieve-Nartey/autonomous-data-quality-agent

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 8 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Validação automática de datasets antes de analytics ou ML
  • Geração de relatórios de qualidade de dados com priorização de ações
  • Detecção de anomalias com duplo método (Z-score + IQR)
  • Pipelines de limpeza de dados guiados por análise agentica

Veredito

Solução robusta de auditoria autônoma de dados que elimina configuração manual. Cobertura abrangente de dimensões de qualidade e output acionável; adequado para pipelines de dados em produção e exploração exploratória.

README

A tradução para o português deste README ainda está na fila.

Autonomous Data Quality Agent

An agentic AI system that autonomously audits any tabular dataset for data quality issues and produces a prioritised, scored remediation report — with no manual configuration of what to check.


Architecture

Raw Dataset (CSV)
        |
        v
   Agent (claude-sonnet-4-5)
        |
        |-- profile_dataset()             -> Shape, types, null counts, stats
        |-- check_missing_values()        -> Severity-classified null analysis
        |-- check_duplicates()            -> Exact + key-column duplicate detection
        |-- check_outliers()              -> Z-score + IQR dual-method detection
        |-- check_format_violations()     -> Regex validation (email, phone, dates)
        |-- check_referential_integrity() -> Foreign key range validation
        |-- generate_dq_report()          -> Scored Markdown report
        |
        v
dq_report_{dataset}.md  +  PNG charts

Features

  • Fully autonomous — the agent decides which checks to run and in what order based on profiling results
  • 7 specialised quality check tools covering all major DQ dimensions
  • Dual outlier detection (Z-score parametric + IQR non-parametric) for robust coverage
  • Severity classification: CRITICAL / WARNING / INFO per issue
  • 0-100 data quality score with visual progress bar
  • Prioritised remediation plan with actionable fix recommendations
  • Two visualisations: missing value rates chart + outlier box plots

Setup

git clone https://github.com/Genevieve-Nartey/autonomous-data-quality-agent.git
cd autonomous-data-quality-agent

python -m venv .venv
source .venv/bin/activate

pip install -r requirements.txt

Create .env:

ANTHROPIC_API_KEY=sk-ant-your-key-here

Usage

Run the demo (generates dirty data + runs full audit):

python data_quality_agent.py

Audit your own CSV:

from data_quality_agent import load_dataset, run_dq_agent
from pathlib import Path

load_dataset(Path("your_data.csv"))
result = run_dq_agent(dataset_name="your_dataset")
print(result)

Run in Jupyter / Kaggle:

Open autonomous_data_quality_agent.ipynb. Every line is explained with inline comments and markdown cells.

On Kaggle, add your ANTHROPIC_API_KEY via Add-ons → Secrets before running.


Quality Checks

CheckMethodWhat It Finds
Missing valuesisnull().sum()Null/NaN counts with CRITICAL/WARNING thresholds
Duplicatesduplicated()Exact rows and key-column near-duplicates
OutliersZ-score + IQRValues beyond 3σ or Tukey fences
Format violationsCompiled regexMalformed emails, phones, dates
Referential integrityRange checkForeign key values outside valid domain

Output Files

FileDescription
dq_report_{dataset}.mdFull scored remediation report
dq_missing_values.pngColour-coded missing value rate bar chart
dq_outliers.pngBox plots for columns with outliers

Project Structure

autonomous-data-quality-agent/
    data_quality_agent.py                  # Production script
    autonomous_data_quality_agent.ipynb    # Annotated Kaggle notebook
    requirements.txt
    .env
    README.md

Key Concepts Demonstrated

ConceptImplementation
Autonomous workflowAgent profiles data first, then selects appropriate checks
Dual outlier methodsZ-score (normal distribution) + IQR (distribution-free)
Module-level regex compilationre.compile() at import time for performance
Data quality scoringWeighted severity score across all detected issues
Deliberate dirty dataSystematic injection of 7 distinct DQ problem types

Licence

MIT

Ver o original no GitHub

Bom para

Relacionados