convez.ai
MCP ServerDados & AnalyticsAvançado
adityavaish

Igniter – Interface moderna para Spark com IA

adityavaish/igniter

Substitui Spark UI com React moderno, MCP e IA analytics integrada

Ver no GitHub
InicialPluginScore · de 0 a 100

Substitui a UI padrão do Spark e Spark History Server com uma interface React moderna e rápida, MCP server integrado e analytics com IA sobre event logs. Usa DuckDB embarcado para análises sub-segundo sobre milhões de tasks, sem dependências externas além do storage.

Instalação

$npx convez-ai adityavaish/igniter

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Monitorar e debugar jobs Spark com UI moderna e responsiva
  • Integrar AI analytics para investigar skew, spill e GC em produção
  • Expor métricas e logs Spark via MCP para automação com AI clients

Veredito

Ferramenta técnica sólida para engenheiros de dados que trabalham com Spark em escala. MCP server bem implementado + analytics com IA agregam valor real, mas exige familiaridade com Spark internals e infraestrutura de cloud storage.

README

A tradução para o português deste README ainda está na fila.

🔥 Igniter

A modern, drop-in replacement for the Spark UI and Spark History Server.

Igniter is a single standalone container. Point it at a storage account that holds Spark event logs and it gives you:

  1. A modern, simplified, fast UI that covers Spark UI + History Server features.
  2. An MCP server layer for easy integration with AI clients.
  3. Integrated AI analytics in the UI for fast investigation.

The only external dependency is the Spark events storage — there is no separate database or other resource to provision.

Why it's fast

Igniter stream-parses each event log once and lands it into an embedded, in-process DuckDB (columnar/vectorized — like SQLite but analytical). The UI, the MCP tools, and the AI analytics all query that same engine, so aggregations over millions of tasks stay sub-second. DuckDB is a library compiled into the binary — not a server — so the container stays self-contained.

Storage (ABFS / S3 / GCS / local)
        │  stream read + decompress (zstd, lz4, snappy, gzip)
        ▼
   ingest / parser  ──►  DuckDB (embedded, columnar)
                          │
        ┌─────────────────┼──────────────────┐
     REST / WS API     MCP server        AI analytics
        │                                     │
     React SPA ◄──────────────────────────────┘

Tech stack

LayerChoice
BackendRust (axum + tokio), single static binary
StoreDuckDB, embedded in-process
Storage I/Oobject_store (Azure ABFS/Blob, S3, GCS, local)
MCPrmcp (official Rust SDK), tools backed by DuckDB SQL
FrontendReact + TypeScript + Vite + Tailwind, TanStack Query/Table
PackagingMulti-stage Docker → single Linux image

Repository layout

igniter/
  Cargo.toml             workspace
  crates/
    model/               shared domain types + Spark event schema
    store/               DuckDB schema + query helpers (P2)
    ingest/              storage access + decompression + parser (P1)
    mcp/                 MCP server + tools (P5)
    server/              axum API/WS + static SPA host + main binary
  web/                   React + Vite SPA
  Dockerfile

Quickstart (with the bundled sample)

A sample Spark event log lives in examples/events/. Point Igniter at it and open the UI:

npm --prefix web install ; npm --prefix web run build
$env:IGNITER_EVENTS_URI = "$PWD\examples\events"
cargo run
# open http://localhost:8080

You'll see the Daily ETL application with jobs, stages, tasks, executors, a SQL console, and an AI Assistant that surfaces skew / spill / GC / failed-task findings.

Development

Prerequisites: Rust (stable) and Node 20+.

# Backend (serves API on :8080, and the built SPA if present)
cargo run

# Frontend (dev server on :5173, proxies /api and /ws to :8080)
npm --prefix web install
npm --prefix web run dev

Build everything for production:

npm --prefix web install ; npm --prefix web run build
cargo build --release

Run the test suite:

cargo test          # Rust workspace (model, ingest, store, mcp, server)
npm --prefix web run build   # type-checks the SPA

Building on Visual Studio 2026 (preview)

DuckDB is compiled from source (the bundled feature). Standard toolchains — Linux/macOS, the Docker image, and Visual Studio 2022 — build it without any extra configuration. The VS 2026 preview removed stdext::checked_array_iterator and defines _SECURE_SCL=0, which trips DuckDB's bundled fmt. If you hit that, add a local (git-ignored) crates/store/src/duckdb_msvc_compat.h:

#pragma once
#include <yvals.h>
#ifdef _SECURE_SCL
#undef _SECURE_SCL
#endif

and a local (git-ignored) .cargo/config.toml that force-includes it (use your absolute repo path):

[env]
"CXXFLAGS_x86_64-pc-windows-msvc" = "/std:c++17 /I<ABSOLUTE>\\crates\\store\\src /FIduckdb_msvc_compat.h"

REST API

MethodPathDescription
GET/api/health, /api/metaLiveness + build info
GET/api/applicationsList applications
GET/api/applications/:idApplication details
GET/api/applications/:id/jobsJobs
GET/api/applications/:id/stagesStages
GET/api/applications/:id/tasksTasks (?stage_id=&limit=)
GET/api/applications/:id/executorsExecutors
GET/api/applications/:id/summaryAggregated metrics
GET/api/applications/:id/analyzeHeuristic AI findings
POST/api/ai/askQ&A ({app_id?, question})
GET/api/query?sql=Read-only SQL over the store
POST/api/reloadRe-ingest from the events storage
WS/ws/statusLive application-count heartbeats

MCP server

The MCP layer exposes the store to AI clients as tools over stdio (list_applications, get_application, find_slow_stages, detect_skew, query_sql). Run the standalone MCP binary:

$env:IGNITER_EVENTS_URI = "$PWD\examples\events"
cargo run -p igniter-mcp --bin igniter-mcp

Example client configuration (Claude Desktop / any MCP client):

{
  "mcpServers": {
    "igniter": {
      "command": "igniter-mcp",
      "env": { "IGNITER_EVENTS_URI": "abfs://container@account.dfs.core.windows.net/spark-events" }
    }
  }
}

AI analytics

Igniter computes heuristic findings directly from the event data — data skew, memory/disk spill, GC pressure, failed tasks, over-partitioning, and executor imbalance — each with concrete metrics and a remediation suggestion. These power the in-UI AI Assistant and the analyze / ask endpoints. If an LLM is configured (OPENAI_API_KEY [+ OPENAI_BASE_URL/OPENAI_MODEL], or AZURE_OPENAI_ENDPOINT + AZURE_OPENAI_API_KEY + AZURE_OPENAI_DEPLOYMENT), the assistant augments these findings with natural-language answers. With no key, it falls back to a deterministic summary — everything works offline.

Container

docker build -t igniter .
docker run -p 8080:8080 \
  -e IGNITER_EVENTS_URI="abfs://container@account.dfs.core.windows.net/spark-events" \
  igniter

Configuration

All configuration is via environment variables:

VariableDefaultDescription
IGNITER_HOST0.0.0.0Bind address
IGNITER_PORT8080Bind port
IGNITER_EVENTS_URI(none)Spark events storage location (required)
IGNITER_LOGS_URIIGNITER_EVENTS_URIStorage prefix for archived driver/executor logs
IGNITER_WEB_DIRweb/distBuilt SPA directory to serve
IGNITER_LOGinfotracing env filter

Storage backends & authentication

IGNITER_EVENTS_URI accepts local paths and cloud URLs; the backend is inferred from the scheme:

BackendExample IGNITER_EVENTS_URI
LocalC:\data\spark-events or file:///data/spark-events
Azure ADLS / Blobabfss://logs@acct.dfs.core.windows.net/, wasbs://logs@acct.blob.core.windows.net/, az://logs/, or a portal https://acct.blob.core.windows.net/logs/sub/ URL
Amazon S3s3://bucket/spark-events/
Google Cloud Storagegs://bucket/spark-events/

Compression (.gz/.zstd/.lz4/.snappy) and Spark's rolling event-log v2 layout (eventlog_v2_*/events_*.zstd, including multi-frame zstd) are handled automatically.

Cloud credentials come from standard object_store env vars:

  • Azure: AZURE_USE_AZURE_CLI=true (reuse your az login), or AZURE_STORAGE_ACCOUNT_NAME + AZURE_STORAGE_ACCOUNT_KEY, or AZURE_STORAGE_SAS_KEY, or a service principal (AZURE_STORAGE_CLIENT_ID/_CLIENT_SECRET/_TENANT_ID), or managed identity.
  • AWS: AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY + AWS_REGION (+ AWS_SESSION_TOKEN, AWS_ENDPOINT).
  • GCS: GOOGLE_SERVICE_ACCOUNT (path to JSON key) or GOOGLE_SERVICE_ACCOUNT_KEY.

Executor & driver logs (stdout/stderr)

Spark event logs do not contain stdout/stderr content — that lives on the cluster and depends on the deployment:

Deploymentstdout/stderr location
YARNnode …/container-logs/<app>/<container>/, or HDFS /tmp/logs/… after aggregation
Kubernetespod logs (kubectl logs), unless shipped to a logging backend
Standaloneworker $SPARK_WORKER_DIR/<app-id>/{stdout,stderr}
Driver (persisted)spark.driver.log.dfsDir — often the same blob storage as event logs

The Logs tab shows stdout/stderr for the driver and each executor, from two kinds of sources:

  • Storage (kind: storage): archived log files discovered under IGNITER_LOGS_URI (defaults to IGNITER_EVENTS_URI). These are read and decompressed directly from blob storage by the backend, so they work even after the cluster is gone. Set IGNITER_LOGS_URI to wherever your driver/ executor logs are persisted (e.g. spark.driver.log.dfsDir). Igniter matches files that reference the application id and look like stdout/stderr/driver logs.
  • URL (kind: url): the per-executor "Log Urls" captured from the event log. These point at live cluster endpoints and only resolve while that cluster is running.

Content is fetched via GET /api/applications/:id/logs (discovery) and GET /api/logs?key=… (storage) / GET /api/logs?url=… (live), with size caps and timeouts. When no sources are found, the tab explains why.

Example (Azure, local dev with the CLI session):

$env:IGNITER_EVENTS_URI = "https://acct.blob.core.windows.net/spark-events/logs/"
$env:AZURE_USE_AZURE_CLI = "true"
.\target\debug\igniter.exe

Call POST /api/reload to re-scan after new logs land.

Status

Functional end-to-end. The Rust workspace and React SPA build, the server ingests Spark event logs into the embedded DuckDB store and serves the REST API, WebSocket status, MCP tools, AI analytics, and the UI. Verified against the bundled sample in examples/events/. See plan.md for the roadmap and history.

License

MIT

Ver o original no GitHub

Bom para

Relacionados