Igniter – Interface moderna para Spark com IA
adityavaish/igniter
Substitui Spark UI com React moderno, MCP e IA analytics integrada
Ver no GitHubSubstitui a UI padrão do Spark e Spark History Server com uma interface React moderna e rápida, MCP server integrado e analytics com IA sobre event logs. Usa DuckDB embarcado para análises sub-segundo sobre milhões de tasks, sem dependências externas além do storage.
Instalação
npx convez-ai adityavaish/igniter1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Monitorar e debugar jobs Spark com UI moderna e responsiva
- Integrar AI analytics para investigar skew, spill e GC em produção
- Expor métricas e logs Spark via MCP para automação com AI clients
Veredito

Ferramenta técnica sólida para engenheiros de dados que trabalham com Spark em escala. MCP server bem implementado + analytics com IA agregam valor real, mas exige familiaridade com Spark internals e infraestrutura de cloud storage.
README
A tradução para o português deste README ainda está na fila.
🔥 Igniter
A modern, drop-in replacement for the Spark UI and Spark History Server.
Igniter is a single standalone container. Point it at a storage account that holds Spark event logs and it gives you:
- A modern, simplified, fast UI that covers Spark UI + History Server features.
- An MCP server layer for easy integration with AI clients.
- Integrated AI analytics in the UI for fast investigation.
The only external dependency is the Spark events storage — there is no separate database or other resource to provision.
Why it's fast
Igniter stream-parses each event log once and lands it into an embedded, in-process DuckDB (columnar/vectorized — like SQLite but analytical). The UI, the MCP tools, and the AI analytics all query that same engine, so aggregations over millions of tasks stay sub-second. DuckDB is a library compiled into the binary — not a server — so the container stays self-contained.
Storage (ABFS / S3 / GCS / local)
│ stream read + decompress (zstd, lz4, snappy, gzip)
▼
ingest / parser ──► DuckDB (embedded, columnar)
│
┌─────────────────┼──────────────────┐
REST / WS API MCP server AI analytics
│ │
React SPA ◄──────────────────────────────┘
Tech stack
| Layer | Choice |
|---|---|
| Backend | Rust (axum + tokio), single static binary |
| Store | DuckDB, embedded in-process |
| Storage I/O | object_store (Azure ABFS/Blob, S3, GCS, local) |
| MCP | rmcp (official Rust SDK), tools backed by DuckDB SQL |
| Frontend | React + TypeScript + Vite + Tailwind, TanStack Query/Table |
| Packaging | Multi-stage Docker → single Linux image |
Repository layout
igniter/
Cargo.toml workspace
crates/
model/ shared domain types + Spark event schema
store/ DuckDB schema + query helpers (P2)
ingest/ storage access + decompression + parser (P1)
mcp/ MCP server + tools (P5)
server/ axum API/WS + static SPA host + main binary
web/ React + Vite SPA
Dockerfile
Quickstart (with the bundled sample)
A sample Spark event log lives in examples/events/. Point Igniter at it and
open the UI:
npm --prefix web install ; npm --prefix web run build
$env:IGNITER_EVENTS_URI = "$PWD\examples\events"
cargo run
# open http://localhost:8080
You'll see the Daily ETL application with jobs, stages, tasks, executors,
a SQL console, and an AI Assistant that surfaces skew / spill / GC / failed-task
findings.
Development
Prerequisites: Rust (stable) and Node 20+.
# Backend (serves API on :8080, and the built SPA if present)
cargo run
# Frontend (dev server on :5173, proxies /api and /ws to :8080)
npm --prefix web install
npm --prefix web run dev
Build everything for production:
npm --prefix web install ; npm --prefix web run build
cargo build --release
Run the test suite:
cargo test # Rust workspace (model, ingest, store, mcp, server)
npm --prefix web run build # type-checks the SPA
Building on Visual Studio 2026 (preview)
DuckDB is compiled from source (the bundled feature). Standard toolchains —
Linux/macOS, the Docker image, and Visual Studio 2022 — build it without any
extra configuration. The VS 2026 preview removed stdext::checked_array_iterator
and defines _SECURE_SCL=0, which trips DuckDB's bundled fmt. If you hit that,
add a local (git-ignored) crates/store/src/duckdb_msvc_compat.h:
#pragma once
#include <yvals.h>
#ifdef _SECURE_SCL
#undef _SECURE_SCL
#endif
and a local (git-ignored) .cargo/config.toml that force-includes it (use your
absolute repo path):
[env]
"CXXFLAGS_x86_64-pc-windows-msvc" = "/std:c++17 /I<ABSOLUTE>\\crates\\store\\src /FIduckdb_msvc_compat.h"
REST API
| Method | Path | Description |
|---|---|---|
| GET | /api/health, /api/meta | Liveness + build info |
| GET | /api/applications | List applications |
| GET | /api/applications/:id | Application details |
| GET | /api/applications/:id/jobs | Jobs |
| GET | /api/applications/:id/stages | Stages |
| GET | /api/applications/:id/tasks | Tasks (?stage_id=&limit=) |
| GET | /api/applications/:id/executors | Executors |
| GET | /api/applications/:id/summary | Aggregated metrics |
| GET | /api/applications/:id/analyze | Heuristic AI findings |
| POST | /api/ai/ask | Q&A ({app_id?, question}) |
| GET | /api/query?sql= | Read-only SQL over the store |
| POST | /api/reload | Re-ingest from the events storage |
| WS | /ws/status | Live application-count heartbeats |
MCP server
The MCP layer exposes the store to AI clients as tools over stdio
(list_applications, get_application, find_slow_stages, detect_skew,
query_sql). Run the standalone MCP binary:
$env:IGNITER_EVENTS_URI = "$PWD\examples\events"
cargo run -p igniter-mcp --bin igniter-mcp
Example client configuration (Claude Desktop / any MCP client):
{
"mcpServers": {
"igniter": {
"command": "igniter-mcp",
"env": { "IGNITER_EVENTS_URI": "abfs://container@account.dfs.core.windows.net/spark-events" }
}
}
}
AI analytics
Igniter computes heuristic findings directly from the event data — data skew,
memory/disk spill, GC pressure, failed tasks, over-partitioning, and executor
imbalance — each with concrete metrics and a remediation suggestion. These power
the in-UI AI Assistant and the analyze / ask endpoints. If an LLM is
configured (OPENAI_API_KEY [+ OPENAI_BASE_URL/OPENAI_MODEL], or
AZURE_OPENAI_ENDPOINT + AZURE_OPENAI_API_KEY + AZURE_OPENAI_DEPLOYMENT),
the assistant augments these findings with natural-language answers. With no key,
it falls back to a deterministic summary — everything works offline.
Container
docker build -t igniter .
docker run -p 8080:8080 \
-e IGNITER_EVENTS_URI="abfs://container@account.dfs.core.windows.net/spark-events" \
igniter
Configuration
All configuration is via environment variables:
| Variable | Default | Description |
|---|---|---|
IGNITER_HOST | 0.0.0.0 | Bind address |
IGNITER_PORT | 8080 | Bind port |
IGNITER_EVENTS_URI | (none) | Spark events storage location (required) |
IGNITER_LOGS_URI | IGNITER_EVENTS_URI | Storage prefix for archived driver/executor logs |
IGNITER_WEB_DIR | web/dist | Built SPA directory to serve |
IGNITER_LOG | info | tracing env filter |
Storage backends & authentication
IGNITER_EVENTS_URI accepts local paths and cloud URLs; the backend is inferred
from the scheme:
| Backend | Example IGNITER_EVENTS_URI |
|---|---|
| Local | C:\data\spark-events or file:///data/spark-events |
| Azure ADLS / Blob | abfss://logs@acct.dfs.core.windows.net/, wasbs://logs@acct.blob.core.windows.net/, az://logs/, or a portal https://acct.blob.core.windows.net/logs/sub/ URL |
| Amazon S3 | s3://bucket/spark-events/ |
| Google Cloud Storage | gs://bucket/spark-events/ |
Compression (.gz/.zstd/.lz4/.snappy) and Spark's rolling event-log v2
layout (eventlog_v2_*/events_*.zstd, including multi-frame zstd) are handled
automatically.
Cloud credentials come from standard object_store env vars:
- Azure:
AZURE_USE_AZURE_CLI=true(reuse youraz login), orAZURE_STORAGE_ACCOUNT_NAME+AZURE_STORAGE_ACCOUNT_KEY, orAZURE_STORAGE_SAS_KEY, or a service principal (AZURE_STORAGE_CLIENT_ID/_CLIENT_SECRET/_TENANT_ID), or managed identity. - AWS:
AWS_ACCESS_KEY_ID+AWS_SECRET_ACCESS_KEY+AWS_REGION(+AWS_SESSION_TOKEN,AWS_ENDPOINT). - GCS:
GOOGLE_SERVICE_ACCOUNT(path to JSON key) orGOOGLE_SERVICE_ACCOUNT_KEY.
Executor & driver logs (stdout/stderr)
Spark event logs do not contain stdout/stderr content — that lives on the cluster and depends on the deployment:
| Deployment | stdout/stderr location |
|---|---|
| YARN | node …/container-logs/<app>/<container>/, or HDFS /tmp/logs/… after aggregation |
| Kubernetes | pod logs (kubectl logs), unless shipped to a logging backend |
| Standalone | worker $SPARK_WORKER_DIR/<app-id>/{stdout,stderr} |
| Driver (persisted) | spark.driver.log.dfsDir — often the same blob storage as event logs |
The Logs tab shows stdout/stderr for the driver and each executor, from two kinds of sources:
- Storage (
kind: storage): archived log files discovered underIGNITER_LOGS_URI(defaults toIGNITER_EVENTS_URI). These are read and decompressed directly from blob storage by the backend, so they work even after the cluster is gone. SetIGNITER_LOGS_URIto wherever your driver/ executor logs are persisted (e.g.spark.driver.log.dfsDir). Igniter matches files that reference the application id and look likestdout/stderr/driver logs. - URL (
kind: url): the per-executor "Log Urls" captured from the event log. These point at live cluster endpoints and only resolve while that cluster is running.
Content is fetched via GET /api/applications/:id/logs (discovery) and
GET /api/logs?key=… (storage) / GET /api/logs?url=… (live), with size caps
and timeouts. When no sources are found, the tab explains why.
Example (Azure, local dev with the CLI session):
$env:IGNITER_EVENTS_URI = "https://acct.blob.core.windows.net/spark-events/logs/"
$env:AZURE_USE_AZURE_CLI = "true"
.\target\debug\igniter.exe
Call POST /api/reload to re-scan after new logs land.
Status
Functional end-to-end. The Rust workspace and React SPA build, the server
ingests Spark event logs into the embedded DuckDB store and serves the REST API,
WebSocket status, MCP tools, AI analytics, and the UI. Verified against the
bundled sample in examples/events/. See plan.md for the roadmap and history.
License
MIT