convez.ai
MCP ServerDevOps & InfraestruturaAvançado
eggrollofchaos

SmartGridBench: Benchmark de Agentes IA para Manutenção de Transformadores

eggrollofchaos/hpml-assetopsbench-smart-grid-mcp

Benchmark industrial de agentes IA para operações de smart grid com MCP

Ver no GitHub
InicialPluginScore · de 0 a 100

MCP server que estende o AssetOpsBench da IBM com um novo domínio de smart grids para validar agentes LLM em cenários de manutenção de transformadores de potência. Integra quatro domínios de ferramentas (IoT, TSFM, FMSR, WO) via protocolo MCP e perfila otimizações de inferência com Llama-3.1-8B em vLLM.

Instalação

$npx convez-ai eggrollofchaos/hpml-assetopsbench-smart-grid-mcp

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 2 dias
Versão
—
Issues abertas
0
Contribuidores
5

Casos de uso

  • Pesquisa e validação de agentes LLM em tarefas industriais reais
  • Benchmarking de diferentes paradigmas de orquestração de ferramentas (Agent-as-Tool vs Plan-Execute)
  • Otimização de pipelines de inferência com profiling end-to-end e quantização

Veredito

Projeto acadêmico robusto de benchmarking industrial com foco em profiling e otimização de agentes LLM via MCP. Bem documentado com datasets validados e experiências rastreadas em W&B, mas de nicho específico para pesquisadores em IA industrial e operações de infraestrutura crítica.

README

A tradução para o português deste README ainda está na fila.


status: canonical-index scope: team-repo owner: Team 13 canonical: true

HPML Final Project: MCP-Based Industrial Agent Benchmarking for Smart Grid Operations

Course: COMS E6998: High Performance Machine Learning Institution: Columbia University Semester: Spring 2026 Instructor: Dr. Kaoutar El Maghraoui Mentor: Dr. Dhaval Patel, IBM Research


Team Information

  • Team Name: Team 13 / District 1101 (SmartGridBench)
  • Members:
    • Akshat Bhandari (ab6174) — LLM / eval harness
    • Aaron Fan (af3623) — EE / systems / Slurm
    • Tanisha Rathod (tr2828) — distributed systems / MCP
    • Wei Alexander Xin (wax1) — PM / data / Agent-as-Tool orchestration
  • Mentor: Dr. Dhaval Patel, IBM Research

Submission

1. Problem Statement

This project extends IBM's AssetOpsBench industrial AI agent benchmark (467 scenarios across 6 HuggingFace subsets) by adding a 7th domain -- Smart Grid power transformers:

  1. Creating 36 paper-grade validated maintenance scenarios for Smart Grid transformers using public-safe synthetic telemetry data (the canonical corpus used for NeurIPS and CourseWorks claims; 25 additional scenarios added post-submission via PR #199 are present in data/scenarios/ but excluded from paper claims)
  2. Wrapping four AssetOpsBench tool domains (IoT, TSFM, FMSR, WO) as MCP servers
  3. Profiling and optimizing the LLM agent inference pipeline when operating through MCP
  4. Comparing two orchestration paradigms (Agent-as-Tool vs Plan-Execute) on end-to-end multi-domain scenarios

2. Model/Application Description

We serve Llama-3.1-8B-Instruct via vLLM, profile end-to-end with PyTorch Profiler, apply 2-3 optimization techniques (INT8 quantization, KV-cache tuning, batched tool-call scheduling), and compare MCP-mediated tool calling against direct function calls to quantify protocol overhead. All experiments tracked with WandB.

ComponentProject implementation
Base modelLlama-3.1-8B-Instruct served through vLLM; selected WatsonX Llama-3.3-70B spot checks
ApplicationIndustrial smart-grid maintenance benchmark extension for AssetOpsBench
Tool interfaceFour Smart Grid MCP servers: IoT, TSFM, FMSR, and work orders
Orchestration variantsAgent-as-Tool, Plan-Execute, Verified PE, and Self-Ask follow-ons
Profiling stackPyTorch Profiler, nvidia-smi telemetry, W&B run metadata, and committed result tables

Datasets

DatasetRowsPrimary AgentLicenseSource
Power Transformers FDD & RUL3,000 files x 420TSFM, IoTCC0Kaggle
DGA Fault Classification201FMSRCC0Kaggle
Transformer Health Index470FMSRODbLKaggle
Current & Voltage Monitoring19,352IoT, TSFM© AuthorsKaggle
Smart Grid Fault Records506WOCC0Kaggle

The repo's tracked data/processed/ artifacts are kept synthetic/public-safe. Restricted-source Kaggle joins remain a local benchmarking path, not a tracked publication path.

Why Llama-3.1-8B-Instruct?

ModelParamsFP16 VRAMTool-calling supportFit for this project
Llama-3.1-8B-Instruct8B~16GBGood, well-documented, strong vLLM supportBest -- fits A6000, enables rapid iteration, INT8 optimization is meaningful (16→8GB)
Phi-4-14B14B~28GBStrong reasoning, less proven for tool callingReasonable but less community tooling
Mistral-Small-24B24B~48GBGoodNeeds A100, fewer experiment iterations per dollar
Gemma-3-27B27B~54GBGoodNeeds A100 80GB, overkill for benchmarking

We select Llama-3.1-8B-Instruct for its favorable profiling characteristics: it fits comfortably on an A6000 with room for KV-cache experiments, and INT8 quantization produces a meaningful memory reduction. We optionally compare against Llama-3.3-70B (AssetOpsBench's default model) via WatsonX API to assess scaling effects.

3. Final Results Summary

The final package is anchored by the submitted report, slide deck, W&B dashboard, and committed metrics under results/metrics/.

Result areaEvidence
Scenario corpusCurrent repo contains 61 SmartGridBench scenario files plus 5 negative validation fixtures; the final paper/result claims preserve the paper-grade frozen scope described in the result tables and report.
MCP transport benchmarkFirst-capture AaT runs show direct, MCP baseline, and optimized MCP paths in results/metrics/notebook02_latency_summary.csv.
Orchestration comparisonVerified PE + Self-Ask was the strongest first-capture PE-family row in the notebook export: 5/6 judge pass rate and 0.833 mean judge score in results/metrics/notebook03_self_ask_ablation.csv.
Full judged floorThe post-PR175 31-scenario aggregate is summarized in results/metrics/gcp_post175_core31_summary.csv.
Profiling evidenceW&B-linked and profiler-linked run inventory is in results/metrics/profiling_inventory.csv.

4. Repository Structure

.
├── README.md                     # This file - project overview, submission links, reproducibility
├── LICENSE                       # MIT license
├── requirements.txt              # Python dependencies (ibm-watsonx-ai, pandas, etc)
├── .github/workflows/            # CI (Black formatting check)
├── deliverables/                 # CourseWorks final report/deck files uploaded with the submission
│
├── data/                         # Data pipeline + processed datasets - see data/README.md
│   ├── build_processed.py        #   Downloads + joins 5 Kaggle datasets via synthesized transformer_id
│   ├── generate_synthetic.py     #   Offline synthetic equivalent (no Kaggle needed)
│   ├── processed/                #   6 public-safe synthetic CSVs tracked in git (asset_metadata, dga_records, …)
│   ├── scenarios/                #   AssetOpsBench-format scenario files - see data/scenarios/README.md
│   ├── knowledge/                #   Structured standards artifacts for PS B generation - see data/knowledge/README.md
│   └── raw/                      #   GITIGNORED raw Kaggle downloads
│
├── mcp_servers/                  # 4 MCP servers on shared base - see mcp_servers/README.md
│   ├── base.py                   #   Shared data loader + utilities
│   ├── iot_server/               #   Sensor reads, asset metadata
│   ├── fmsr_server/              #   Failure search, IEC 60599 Rogers Ratio DGA analysis
│   ├── tsfm_server/              #   RUL forecast, z-score anomaly, OLS trend
│   └── wo_server/                #   Work order CRUD, downtime estimation
│
├── scripts/                      # Utility scripts - see scripts/README.md
│   ├── verify_watsonx.py         #   WatsonX access verification + latency benchmarking
│   ├── run_experiment.sh         #   Canonical SmartGridBench runner (PE baseline + Self-Ask/Verified PE follow-ons)
│   ├── setup_insomnia.sh         #   Shared Insomnia environment setup
│   ├── vllm_serve.sh             #   Self-hosted vLLM serve + smoke path on Insomnia
│   ├── test_inference.sh         #   Sanity checks against a live vLLM endpoint
│   └── benchmark_prompts/        #   Prompt templates for latency tests
│
├── benchmarks/                   # Raw latency/throughput/judge runs - see benchmarks/README.md
│   ├── cell_A_direct/            #   Direct-tool baseline artifacts
│   ├── cell_B_mcp_baseline/      #   MCP baseline artifacts
│   ├── cell_C_mcp_optimized/     #   Optimized MCP path artifacts
│   ├── cell_Y_plan_execute/      #   Plan-Execute proof path (WatsonX smoke landed)
│   └── cell_Z_hybrid/            #   Optional third-method slot (Verified PE preferred over generic Hybrid)
│
├── notebooks/                    # Jupyter notebooks - see notebooks/README.md
├── profiling/                    # PyTorch Profiler + Nsight - see profiling/README.md
├── results/                      # Curated metrics + figures - see results/README.md
│
├── docs/                         # Living authored documentation - see docs/README.md
│   └── knowledge-base/reference/               #   Lower-churn class / mentor / setup reference docs
├── planning/                     # Meeting agendas + working notes
└── reports/                      # Build notes, historical submissions, and report work products - see reports/README.md
    └── archive/                  #   Superseded drafts

5. Reproducibility Instructions

A. Environment Setup

Prerequisites

  • Python 3.12+
  • Docker (for CouchDB)
  • CUDA-capable GPU (16GB+ VRAM recommended)
  • Access to Columbia Insomnia cluster or Google Cloud GPU instance
  • WatsonX API key (via Codabench)

Installation

git clone https://github.com/HPML6998-S26-Team13/hpml-assetopsbench-smart-grid-mcp.git
cd hpml-assetopsbench-smart-grid-mcp
uv venv .venv
source .venv/bin/activate
uv pip install -r requirements.txt

Optional overlays:

# interactive notebook authoring
uv pip install -r requirements-notebooks.txt

# Insomnia / cluster serving stack
uv pip install -r requirements-insomnia.txt

requirements.txt now also includes the portable AssetOpsBench PE-client dependencies (litellm and mcp[cli]) needed by the repo-local Self-Ask PE / Verified PE runners. The cluster overlay in requirements-insomnia.txt layers vLLM and CUDA-specific pins on top of that base.

B. Experiment Tracking Dashboard

C. Dataset

The repo tracks public-safe synthetic processed data under data/processed/ and AssetOpsBench-format Smart Grid scenarios under data/scenarios/. Raw Kaggle downloads remain gitignored under data/raw/ and are not required to inspect the submitted results.

python data/generate_synthetic.py

D. Training

This project benchmarks inference and tool orchestration; it does not fine-tune the base Llama model. The model-serving path is configured through vLLM, WatsonX-compatible LiteLLM endpoints, and the configs/*.env experiment files.

E. Evaluation

Judge outputs and summarized metrics are committed under results/. To score a new completed run directory, use the judge helper with the run directory and matching output path:

python scripts/judge_trajectory.py \
  --run-dir benchmarks/cell_B_mcp_baseline/raw/<run-id> \
  --scenario-dir data/scenarios \
  --out results/metrics/scenario_scores.jsonl \
  --log-dir results/judge_logs

F. Profiling

Profiling is enabled through the runner environment and summarized in results/metrics/profiling_inventory.csv.

ENABLE_WANDB=1 TORCH_PROFILE=1 bash scripts/run_experiment.sh configs/example_baseline.env
python scripts/build_profiling_inventory.py

G. Quickstart: Reproduce the Baseline Runner Path

For the currently proven benchmark-facing path:

bash scripts/run_experiment.sh configs/example_baseline.env

See:

6. Results and Observations

  • The AaT direct, MCP baseline, and optimized MCP latency comparison is exported in results/metrics/notebook02_latency_summary.csv.
  • The first-capture orchestration and Self-Ask comparison is exported in results/metrics/notebook03_self_ask_ablation.csv.
  • The final submitted claim set is backed by committed result tables, W&B run links, benchmark artifacts, and exported report/deck deliverables.

Historical Status Snapshot

Last updated: Apr 18, 2026 - Apr 16 post-call audit resolved the WO architecture decision, closed the first-WandB-run milestone, and narrowed the active risk to overdue W2 closeout plus W3 profiling / PS B execution.

Week 1 (complete):

  • Problem statement finalized (four contributions)
  • Research proposal drafted and shared with mentor via Overleaf; mentor endorsed NeurIPS 2026 Datasets & Benchmarks track
  • GitHub repo scaffolded, WandB team created, repo now public (Apr 7)
  • 5 Kaggle datasets identified, AssetOpsBench forked and reviewed
  • Compute confirmed (Insomnia cluster + GCP credits) and compute plan committed (docs/archive/compute_plan.md)
  • WatsonX API access received from mentor (Apr 5) and verified end-to-end - 6 Llama models available; Llama-4-Maverick-17B and Llama-3.3-70B-instruct benchmarked (docs/knowledge-base/reference/2026-04-06_watsonx_access.md)
  • Data pipeline + tracked public-safe processed datasets landed (data/processed/ with synthetic asset metadata, DGA records, failure modes, fault records, RUL labels, and sensor readings - development-ready and safe to publish)
  • MCP server skeletons landed for all four domains (IoT, FMSR, TSFM, WO) on a shared base class with substantive domain logic (IEC 60599 Rogers Ratio DGA analysis, RUL forecast, anomaly detection, work-order CRUD)
  • docs/data_pipeline.tex paper section drafted
  • Mid-point report submitted (reports/2026-04-06_midpoint_submission.pdf) to Courseworks on Mon Apr 6

Week 2 (landed on canonical history):

  • GitHub Projects reset as the canonical planning surface, with weekly iterations, workstream parent issues, and delivery-gate milestones
  • Successful Insomnia A6000 vLLM serve smoke test for Llama-3.1-8B-Instruct, with kept smoke-test logs and fixed serve/test scripts
  • First real SmartGridBench WandB run is live and back-linked to committed benchmark artifacts
  • Plan-Execute is wired to the team’s Smart Grid MCP servers as a real experiment condition, with a successful WatsonX-hosted 70B / Mac 1-scenario smoke proof run under benchmarks/cell_Y_plan_execute/
  • Scenario realism validation note landed with IEEE / IEC grounded findings for Dhaval-facing review

Still open from the original W2 critical path / backlog:

  • #3 Canonical benchmark scenario proof on the AssetOpsBench stack (Akshat)
  • #58 Benchmark-Llama-path validation closeout plus #9-#12 MCP hardening/tests (Tanisha) - PR #115 now contains a real Insomnia A6000 / self-hosted 8B / all-4-server proof run plus branch-specific long-context serve notes, but the merge/readme cleanup is still open (see docs/insomnia_runbook.md)
  • #7 / #59 profiling wrappers and the first profiling-linked experiment capture path (Aaron)
  • #15 / #17 / #18 / #20 scenario-count, judge, and first trajectory artifacts (Akshat)

Paper-grade scenario corpus: 36 validated SmartGridBench scenarios — 31 hand-authored + 5 promoted generated. This is the canonical count used for the final NeurIPS and CourseWorks claims (frozen at NeurIPS submission, 2026-05-07).

Current repo corpus: 61 scenario files under data/scenarios/ plus 5 negative validation fixtures. The 25 scenarios beyond the 36 paper-grade canonical were added post-submission via PR #199 (#55 batch A and B) to grow the repo corpus and are deliberately not part of any evaluated/judged claims.

Resolved during the Apr 16 post-call audit:

  • #13 WO architecture review closed as a documented keep-vs-pivot decision
  • #28 closed against the first real shared WandB run and its committed benchmark artifacts

W3 focus (Apr 14-20):

  • Experiment 1 profiling captures plus profiling-to-WandB linkage (#25, #27)
  • Problem Statement B kickoff: Knowledge Plugin, first generation prototype, and evaluation methodology (#50, #2, #51)
  • NeurIPS abstract outline and title candidates (#77)
  • Runbook consolidation for the infra / serve / profiling path (#37)

Committed W3-W5 tracks:

  • Experiment 1: MCP overhead and optimization (Direct vs MCP-baseline vs MCP-optimized)
  • Experiment 2: orchestration comparison and failure analysis
  • Problem Statement B extension: scenario generation pipeline, Knowledge Plugin, and validation methodology
  • NeurIPS 2026 draft first, then back-port to the class IEEE report format

Current default scope decision: the working comparison is still vanilla Agent-as-Tool vs vanilla Plan-Execute. The repo now also has an active local mitigation stream for PE + Self-Ask and an optional Verified PE third-method prototype, but neither should muddy the honesty of the core AaT vs PE story. The primary local benchmark model remains self-hosted Llama-3.1-8B-Instruct with 70B reserved for selective WatsonX spot-checks.

7. Notes

Repository Navigation

Key Dates

DateMilestone
Mon Apr 6Mid-point report due (Courseworks, 11:59pm)
Tue May 5NeurIPS 2026 Datasets & Benchmarks abstract submitted
Thu May 7NeurIPS 2026 full submission submitted
Thu May 7Final class presentation delivered
Sun May 10CourseWorks final report/package due (extended deadline)

References and Resources

See also: docs/knowledge-base/reference/2026-04-05_project_reference.md for class requirements, grading, and mentor guidance.

AI Use Disclosure

Per the HPML AI Use Policy, Team 13 used AI assistance during this project.

Tools used: Claude and Codex.

Specific purpose: We used AI tools for brainstorming, debugging support, code navigation, runbook/checklist assistance, and prose polishing. Experimental design decisions, reported measurements, profiling interpretations, quantitative analysis, and final claims were checked by team members against committed code, experiment logs, dashboard records, and exported result artifacts.

Sections affected: README/runbook wording, report and presentation polishing, checklist maintenance, and selected implementation/debugging workflows.

How we verified correctness: We ran the reported experiments ourselves, preserved result artifacts under benchmarks/, results/, and reports/, cross-checked headline claims against the committed evidence tables, and manually reviewed final public-facing wording before submission.

By submitting this project, the team confirms that the analysis, interpretations, and conclusions are our own, and that AI assistance is disclosed here.

License

MIT; see LICENSE.

Citation

If citing this course project, use:

SmartGridBench Team 13. "MCP-Based Industrial Agent Benchmarking for Smart Grid Operations." COMS E6998 High Performance Machine Learning Final Project, Columbia University, Spring 2026.

Contact

For project questions, use the GitHub issue tracker or contact the Team 13 members listed above.

Ver o original no GitHub

Bom para

Relacionados