iceberg-mcp-data
dragonejt/iceberg-mcp-data
Data Pipeline for Iceberg MCP Server
Instalação
npx convez-ai dragonejt/iceberg-mcp-data1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
README
A tradução para o português deste README ainda está na fila.
iceberg-mcp-data
Pipeline Overview
iceberg-mcp-data is a data pipeline that ingests PyPI file downloads and distribution metadata for iceberg-mcp-server, and then performs ETL and data processing on it. This data pipeline consists of the following components:
-
Transformation: The
transformations/folder contains the Spark jobs that run using Snowpark Connect for Apache Spark. -
Notebooks: The
notebooks/folder contains Jupyter notebooks that interact with the data on Snowflake using Apache Iceberg and Snowflake Connect for Apache Spark.
Local Development
Local development with iceberg-mcp-data is done using the Snowflake Extension for VS Code.
Building and Running
-
Install all packages and set up the virtual environment by running
uv sync. -
Run the
pipelinecommand to run the entire data pipeline on Snowpark Connect for Apache Spark using your Snowflake connection. This will run all transformation jobs in order. -
Run the
pipeline run-job <job name>to run a specific job.
Testing
This repository uses pytest for test running, although the tests themselves are structured in the unittest format. Running tests involves invoking pytest like any other project. If you use VS Code or a fork for development, the VS Code Python Extension will enable automatic test discovery and running in the Testing sidebar. Tests will also be run with coverage in the integration workflow.
Linting and Formatting
iceberg-mcp-server uses Ruff and ty for linting, formatting, and type checking. The standard commands to run are:
ruff check --fix # linting
ruff format # formatting
ty check # type checking
The Ruff configuration is found in pyproject.toml, and all autofixable issues will be autofixed. If you use VS Code or a fork for development, the VS Code Ruff Extension and VS Code ty Extension will enable viewing issues from Ruff and ty within your editor. Additionally, Ruff, ty, and CodeQL analysis will be run in the integration workflow.
Orchestration
In production, daily GitHub Actions workflows run to orchestrate the entire data pipeline.