convez.ai
Agente
yihui504

TestVDB

yihui504/TestVDB

Ver no GitHub
InicialPluginScore · de 0 a 100

A multi-agent, fully automated vector database testing system in the form of a Claude Code plugin.

Instalação

$npx convez-ai yihui504/TestVDB

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 2 meses
Versão
v2.2.0
Issues abertas
0
Contribuidores
1

README

A tradução para o português deste README ainda está na fila.

TestVDB

English | 中文

License: MIT Claude Code Plugin Version

Automated Defect Mining for Vector Databases

TestVDB is an LLM-powered Claude Code plugin that automatically discovers compliance defects in vector databases. It reverse-engineers structured contracts from official documentation, generates targeted attack scripts through multi-agent debate, executes them in Docker sandboxes, and produces verified defect reports with full evidence chains.

Currently supports Milvus, Qdrant, Weaviate, and pgvector.


What's New in v2.1.1

  • Quality Hardening: All attack scripts now use safe_request() pattern — zero bare API calls, zero script crashes on connection/timeout errors
  • AST-based API Format Validation: New validate_api_format.py in Stage 1 debate performs AST-level checking of attack scripts, rejecting bare .json() chains before execution
  • Reporter Split: reporter.md (defect reports) split from reporter-mre.md (MRE scripts) — separation of concerns, smaller file sizes
  • Code Deduplication: _session_utils.py shared across 7 hook/maintenance scripts, eliminating ~100 lines of duplicate _plugin_root() / is_session_locked() implementations
  • Nested Dispatch Prohibition: Explicit prohibition of nested agent dispatch across all agent prompts — a platform limitation discovered and codified
  • Orchestrator Lifecycle Management: Extracted to orchestrator-lifecycle.md (error handling strategy, PreCompact/PostCompact context protection, progress visibility, multi-DB parallelism)
  • pgvector SQL Endpoint Patterns: Completed strategy_extractor.py with DDL/DML/Search/Index regex patterns for SQL-based vector databases
  • Exception Hardening: Bare except: → specific exception types, --target missing-value validation, .env quote stripping
  • Agent Fleet: 18 agents (+ orchestrator-lifecycle, + reporter-mre), 24 scripts (+ _session_utils.py, + validate_api_format.py)

Full Changelog →


What's New in v2.1

  • Phase 0: Strategic Intelligence Layer: Pre-mining historical defect analysis — crawls target DB GitHub repos for past issues and merged PRs, builds threat models and cognitive blindspot profiles
  • Issue Tri-Classification: Classifies historical issues into positive (acknowledged bugs), negative (by-design / wontfix), and invalid (no response) — extracts root cause patterns from positive samples
  • Developer Cognitive Blindspot Model: 5-category taxonomy (BS-01 through BS-05) mapping systemic developer oversights to attack strategies
  • Cross-DB Bug Shape Migration: Marks extracted bug patterns as cross_db_applicable for Milvus→Qdrant→Weaviate→PGVector strategy reuse
  • 3 New Agents: issue-miner (raw), bug-shape-extractor (redacted), threat-modeler (redacted) — total agent fleet: 18

Full Changelog →


Table of Contents


How It Works

TestVDB operates as a Claude Code plugin with a 7-phase pipeline orchestrated by 18 specialized agents:

Phase 0: Strategic Intelligence      -- Historical issue mining + bug shape extraction + threat modeling
Phase 1: Knowledge Extraction        -- WebSearch + WebFetch official docs
Phase 2: Contract Formalization      -- Structured JSON contract from raw docs
Phase 3: Attack Script Generation    -- 9 concurrent agents (Fan-Out) + Stage 1 debate (inc. AST validation)
Phase 4: Sandbox Execution           -- Dual-tier execution (host Python / Docker stdin pipe)
Phase 5: Defect Judgment             -- 4 judge agents + Stage 2 voting debate
Phase 6: Report Generation           -- Defect reports + MRE scripts + strategy extraction

The pipeline runs iteratively: each round injects reflection_context from the previous round into attack agents, enabling strategy adaptation. Phase 0 intelligence (threat model + cognitive blindspots) is also injected to prioritize attack surfaces with historically high defect density. Stage 1 debate now includes AST-level API format validation. Stalemate detection (5 consecutive rounds with no new defects) triggers strategy re-evaluation.


What's New in v2.0

Fan-Out Attack Trio

Instead of 3 attack agents, v2.0 deploys 9 concurrent agents — each attack type (boundary, state, semantic) × 3 focus profiles:

  • priority_first — targets high-priority constraints first
  • coverage_gap — fills gaps from coverage.json
  • rejection_pattern — reverse-engineers bypass paths from prior round rejections

All 9 agents run in parallel. Results go through 3-tier deduplication (endpoint × constraint × strategy), with independent-verification confidence bonuses.

Cross-Session Strategy Evolution

Mining strategies are now persistent across sessions and DB targets:

  1. strategy_extractor.py extracts successful attack patterns after each session
  2. strategy_registry/ stores strategies per-DB ({db}_strategies.json) and globally (global_strategies.json)
  3. strategy_injector.py reads applicable strategies and injects them into Attack Agent prompts
  4. Cross-DB migration: strategies from Milvus automatically map to Qdrant equivalents

Every evolution is audited in evolution_log.jsonl.

7-Mode AI Failure Checklist

scripts/ai_failure_check.py validates the pipeline output with 7 detection modes:

ModeCheckAction on Failure
M1Script syntax errorsRewind
M2Source URL reachabilityReject
M3Execution result data validationReject
M4.done marker completeness (pipeline integrity)Halt
M5Defect classification consistencyRewind
M6Methodology fabrication detectionReject
M7Infinite loop detectionHalt

Material Passport

Every structured_contract.json includes a _passport field with SHA-256 hash:

{
  "_passport": {
    "schema_version": "2.0",
    "hash_algorithm": "sha256",
    "hash": "88ed0dc...",
    "endpoint_count": 68,
    "constraint_count": 39,
    "core_crud_coverage_pct": 95.0
  }
}

scripts/passport_verify.py validates contract integrity:

  • Exit 0: PASS — hash matches
  • Exit 1: NO_PASSPORT — legacy contract format
  • Exit 2: TAMPERED — hash mismatch, pipeline rejects and re-generates

Data Access Level

Every agent file declares a dataAccess level in frontmatter:

  • raw — access to raw documentation and external network
  • redacted — only access to specific session artifacts
  • verified_only — only access to judge-verified data

Defect Taxonomy

TestVDB classifies discovered defects into four MECE (Mutually Exclusive, Collectively Exhaustive) categories:

TypeNameDefinitionExample
Type 1Illegal SuccessInput violating documented constraints is accepted (2xx instead of 4xx)limit=-1 returns 200 OK
Type 2Poor DiagnosticsInvalid input correctly rejected, but error message is unclearReturns "Unknown Error" instead of "Invalid Dimension"
Type 3Runtime FailureValid input causes crash, 500 error, or abnormal behaviorLegal search request returns 500
Type 4State/Logic ViolationAPI returns success, but internal state is inconsistentINSERT 3 rows, COUNT returns 2

Classification decision tree:

1. Illegal input accepted?     --> Type 1 (Illegal Success)
2. Valid input causes crash?   --> Type 3 (Runtime Failure)
3. Error message unclear?      --> Type 2 (Poor Diagnostics)
4. State/result inconsistent?  --> Type 4 (State/Logic Violation)
5. None of the above           --> Not a defect

Quick Start

1. Install Claude Code CLI

npm install -g @anthropic-ai/claude-code

2. Install TestVDB Plugin

Method A: Claude Code Marketplace (recommended)

/plugin marketplace add yihui504/TestVDB
/plugin install testvdb@yihui504-TestVDB

Method B: Local clone

git clone https://github.com/yihui504/TestVDB.git
claude --plugin-dir TestVDB

3. Mine Defects

Use the /testvdb:mine command inside a Claude Code session:

/testvdb:mine milvus v2.6.17
/testvdb:mine qdrant v1.12.0 --max-rounds 3
/testvdb:mine weaviate 1.25.0 --min-defects 2
/testvdb:mine pgvector pg17 --max-rounds 0

Installation

Marketplace Install (Recommended)

TestVDB is distributed as a Claude Code plugin. Install via the marketplace:

# In any Claude Code session:
/plugin marketplace add yihui504/TestVDB
/plugin install testvdb@yihui504-TestVDB

The plugin installs globally and persists across sessions. Use /help to verify — you should see /testvdb:mine in the command list.

Local Development Install

git clone https://github.com/yihui504/TestVDB.git
cd TestVDB
claude --plugin-dir .

Note: --plugin-dir . loads the plugin for the current session only. File changes take effect in the next session.


Usage

Command Reference

/testvdb:mine <db> <version> [--max-rounds N] [--min-defects N]
ParameterRequiredDefaultDescription
<db>Yes--Target database: milvus, qdrant, weaviate, or pgvector
<version>Yes--Target version (e.g., v2.6.17, v1.12.0, pg17)
--max-rounds NNo5Maximum mining rounds. 0 for unlimited
--min-defects NNo1Minimum defects before early termination

Termination Conditions

The pipeline stops when any of the following is met:

  1. Stalemate: 5 consecutive rounds with no new defects
  2. Coverage: Contract coverage reaches >= 95%
  3. Max Rounds: --max-rounds limit reached
  4. Min Defects: --min-defects threshold reached

Error Recovery

Re-run the same command to resume an interrupted session. The system auto-detects incomplete sessions via checkpoint files.

Multi-DB Parallel Mining

# Terminal 1
/testvdb:mine milvus v2.6.17
# Terminal 2
/testvdb:mine qdrant v1.12.0

Output Structure

Results are written to results/{db}/{version}/{timestamp}/:

results/qdrant/v1.12.0/2026-06-07T15-30-00Z/
  defects/defect-1.md              # Defect report
  mre/defect-1-script.py           # Minimal Reproducible Example
  summary.md                       # Session summary
  debate_logs/stage1.json          # Attack script peer review logs
  debate_logs/stage2.json          # Judge quartet voting logs
  structured_contract.json         # Generated contract (with _passport)
  session_metadata.json            # Session metadata
  coverage.json                    # Endpoint coverage tracking
  experience_handoff.json          # Cross-round reflection context

intelligence/{target}/             # v2.1 Phase 0 strategic intelligence (per-DB, TTL 30d)
  issue_corpus.json                # Raw historical issue corpus
  commit_corpus.json               # Raw historical commit/PR corpus
  classified_issues.json           # Tri-classification results
  bug_shapes.json                  # Extracted root cause patterns
  developer_cognition.json         # Developer cognitive boundary analysis
  threat_model.json                # Threat model + cognitive blindspots

Architecture

Agent Fleet (16 agent types + 2 auxiliary specs)

Note: 16 agent types are registered in plugin.json and dispatchable via Agent(subagent_type="testvdb:xxx"). 2 additional specifications (orchestrator-lifecycle.md, reporter-mre.md) serve as auxiliary references — the former defines lifecycle management rules consumed by the main process, the latter is an agent registered separately for MRE generation.

AgentdataAccessRole
orchestratorredactedPipeline coordinator; dispatches all sub-agents
orchestrator-lifecycleredactedLifecycle management: error handling, Pre/PostCompact, progress visibility (extracted from orchestrator)
issue-minerrawCrawls historical issues and merged PRs from target repos
bug-shape-extractorredactedTri-classifies issues (positive/negative/invalid), extracts root cause patterns
threat-modelerredactedBuilds threat model and cognitive blindspot model from historical data
knowledge-extractorrawCrawls official docs, extracts endpoints/parameters/constraints
contract-formalizerredactedConverts raw knowledge into structured JSON contract with _passport
attack-boundaryredactedGenerates boundary-value attack scripts
attack-stateredactedGenerates state-transition attack scripts
attack-semanticredactedGenerates semantic/logic attack scripts
docker-executorredactedDual-tier script execution (host Python / Docker stdin pipe)
judge-docrawValidates document reference accessibility and content consistency (with network verification)
judge-evidenceverified_onlyValidates evidence chain completeness
judge-noveltyrawChecks defect novelty via GitHub search
judge-severityverified_onlyAssesses defect severity
reporterverified_onlyGenerates defect reports with evidence chains
reporter-mreverified_onlyGenerates self-contained MRE scripts for confirmed defects
model-testredactedCCSwitch tier routing verification

Skills (4 skills)

SkillPurpose
pipeline6-phase pipeline SOP for the orchestrator
contract-schemaJSON schema reference for contract formalization
defect-taxonomyFour-type defect classification reference
docker-templatesDocker container templates for each target DB

2-Stage Debate Mechanism

Stage 1 — Attack Script Peer Review: Attack agents independently generate test scripts. Scripts undergo peer review voting before sandbox execution. Only scripts that pass the vote proceed.

Stage 2 — Judge Quartet Voting: After sandbox execution, the four judge agents independently review results. judge-doc runs first as a weight regulator (DOC_VERIFIED / DOC_PARTIAL / DOC_MISMATCH) adjusting the strictness of the other three judges. A defect is confirmed when evidence and severity both vote is_defect. Novelty always votes is_defect with novelty_rating metadata, not participating in defect confirmation but affecting report priority.

Pre-Submit Reverify Gate

Every confirmed defect is re-verified in a fresh Docker container before report generation. This eliminates false positives caused by container state leakage or transient errors.


Directory Structure

TestVDB/
  .claude-plugin/plugin.json      Plugin manifest (name, version, commands, agents)
  .mcp.json                       MCP server config (GitHub API)
  agents/                         18 agent definitions
    orchestrator.md
    orchestrator-lifecycle.md
    issue-miner.md
    bug-shape-extractor.md
    threat-modeler.md
    knowledge-extractor.md
    contract-formalizer.md
    attack-boundary.md
    attack-state.md
    attack-semantic.md
    docker-executor.md
    judge-doc.md
    judge-evidence.md
    judge-novelty.md
    judge-severity.md
    reporter.md
    reporter-mre.md
    model-test.md
  commands/mine.md                Entry command (/testvdb:mine)
  docker/                         Docker Compose templates
    crawl4ai.yml                  Crawl4AI web scraper service
    milvus.yml                    Milvus (etcd + MinIO + standalone)
    qdrant.yml                    Qdrant standalone
    weaviate.yml                  Weaviate standalone
    pgvector.yml                  PGVector standalone
  hooks/hooks.json                Lifecycle hooks (pre-compact, post-compact, etc.)
  skills/                         4 skill definitions
    pipeline/SKILL.md
    contract-schema/SKILL.md
    defect-taxonomy/SKILL.md
    docker-templates/SKILL.md
  intelligence/                   v2.1 Strategic intelligence cache (per-DB, TTL 30d)
  contracts/                      Reference contracts & schema
    AGENTS.md
    settings_schema.json          Settings validation schema
    pgvector_contract.json        PGVector reference contract
    weaviate_contract.json        Weaviate reference contract
  scripts/                        Infrastructure scripts (24 scripts)
    passport_verify.py            Material Passport hash verification
    strategy_extractor.py         Cross-session strategy extraction
    strategy_injector.py          Cross-DB strategy injection
    ai_failure_check.py           7-mode AI failure checklist
    validate_api_format.py        AST-based API call format validation (v2.1.1)
    _session_utils.py             Shared session utilities (v2.1.1)
    preflight.py                  Session pre-flight checks
    crawl_fetch.py                Crawl4AI web scraper (primary)
    crawl_milvus.py               Milvus-specific doc crawler
    hook_runner.py                Cross-platform hook executor
    github_search.py              GitHub issue/code search
    prioritizer.py                Attack script prioritization
    verify_defects.py             Batch defect verification
    find_python.py                Python interpreter resolution
    developer_attitude.py         Developer sentiment analysis
    cleanup_stop.py               Session cleanup
    emergency_cleanup.py          Emergency container cleanup
    log_execution.py              Execution logging
    notify_check.py               Notification config validation
    postcompact_verify.py         Post-compaction state recovery
    precompact_save.py            Pre-compaction state preservation
    retry_policy.py               Retry policy reporter
    gen_weaviate_contract.py      Weaviate contract generation
    validate_weaviate_contract.py Weaviate contract validation
  docs/                           Documentation
    reviews/                      Code review reports
  settings.json                   Plugin configuration (26+ parameters)
  AGENTS.md                       Agent orchestration rules
  THEORETICAL_FRAMEWORK.md        Research paper
  LICENSE                         MIT License

Configuration

settings.json

Configuration parameters organized into sections:

SectionKey ParametersDescription
dockercleanup_on_exit, startup_timeout_seconds, per-DB portsDocker container lifecycle and port mapping
githubtokenGitHub personal access token for novelty judge
retrymax_attempts, *_delay_secondsRetry and delay policies
pipelinedefault_max_rounds, default_min_defectsPipeline execution limits
resultsbase_dir, max_sessionsOutput directory and session management
knowledgecache_enabled, cache_ttl_hoursContract caching (default: 168h / 7 days)
notificationon_severity, webhook_urlAlert configuration for critical defects
networkproxyHTTP proxy for network requests
evolutionenabled, strategy_registry_dir, max_strategies_per_injection, min_confidence_for_injectionCross-session strategy evolution
fan_outenabled, seeds_per_agent, profilesFan-Out attack dispatch (9 concurrent agents)
ai_failure_checkenabled, halt_on, reject_on, rewind_on7-mode AI failure detection
material_passportenabled, hash_algorithm, reject_on_tamperContract hash integrity verification
intelligenceenabled, cache_ttl_hours, time_window_months, max_issues, max_commits, inject_to_attack_agents, inject_to_judge_agentsv2.1 Phase 0 strategic intelligence config

.mcp.json

Configures the GitHub MCP server used by the novelty judge to search for duplicate issues:

{
  "mcpServers": {
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "${GITHUB_TOKEN}"
      }
    }
  }
}

Requirements

RequirementVersionNotes
LLM ModelClaude Sonnet/OpusRuns via Claude Code
Claude Code CLILatestnpm install -g @anthropic-ai/claude-code
Docker Engine20+Must be running before pipeline start
Python3.9+Used by hooks and helper scripts
Disk Space10GB+For Docker images and results
Docker Hub Token--Recommended. Set DOCKER_HUB_TOKEN env var for higher rate limits
Network Access--WebFetch must reach target doc sites (milvus.io, qdrant.tech, etc.)
GitHub Token--Optional; enables full novelty judge via GitHub API

Testing on Claude Code

Quick Test

# Start Claude Code with the plugin loaded
cd TestVDB
claude --plugin-dir .

# Run a single-round test on Qdrant (simplest setup, single container)
/testvdb:mine qdrant v1.12.0 --max-rounds 1

Results will be in results/qdrant/v1.12.0/<timestamp>/.

Debugging

# Enable debug mode to see plugin loading details
claude --plugin-dir . --debug

# Check loaded agents
/agents

# Check available commands
/help

Evidence Chain Standard

Every confirmed defect must satisfy the 3-ring evidence chain:

  1. Contract Reference: The specific constraint violated, with constraint ID from the structured contract
  2. Source URL: Direct link to the official documentation page that defines the constraint
  3. Documentation Link: (Optional) Source code reference or GitHub issue for additional context

Additionally, each defect report includes a Minimal Reproducible Example (MRE) — a self-contained Python script that can be run in a fresh Docker container to reproduce the defect.


License

This project is licensed under the MIT License.

Ver o original no GitHub

Bom para

Relacionados