convez.ai
MCP Server
dragonejt

iceberg-mcp-data

dragonejt/iceberg-mcp-data

Ver no GitHub
InicialPluginScore · de 0 a 100

Data Pipeline for Iceberg MCP Server

Instalação

$npx convez-ai dragonejt/iceberg-mcp-data

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
1
Forks
0
Acompanhando
0
Saúde
42%
Último commit
há 2 dias
Versão
—
Issues abertas
1
Contribuidores
1

README

A tradução para o português deste README ainda está na fila.

iceberg-mcp-data

codacy coverage integrate

Pipeline Overview

iceberg-mcp-data is a data pipeline that ingests PyPI file downloads and distribution metadata for iceberg-mcp-server, and then performs ETL and data processing on it. This data pipeline consists of the following components:

  1. Transformation: The transformations/ folder contains the Spark jobs that run using Snowpark Connect for Apache Spark.

  2. Notebooks: The notebooks/ folder contains Jupyter notebooks that interact with the data on Snowflake using Apache Iceberg and Snowflake Connect for Apache Spark.

Local Development

Local development with iceberg-mcp-data is done using the Snowflake Extension for VS Code.

Building and Running

  1. Install all packages and set up the virtual environment by running uv sync.

  2. Run the pipeline command to run the entire data pipeline on Snowpark Connect for Apache Spark using your Snowflake connection. This will run all transformation jobs in order.

  3. Run the pipeline run-job <job name> to run a specific job.

Testing

This repository uses pytest for test running, although the tests themselves are structured in the unittest format. Running tests involves invoking pytest like any other project. If you use VS Code or a fork for development, the VS Code Python Extension will enable automatic test discovery and running in the Testing sidebar. Tests will also be run with coverage in the integration workflow.

Linting and Formatting

iceberg-mcp-server uses Ruff and ty for linting, formatting, and type checking. The standard commands to run are:

ruff check --fix # linting
ruff format # formatting
ty check # type checking

The Ruff configuration is found in pyproject.toml, and all autofixable issues will be autofixed. If you use VS Code or a fork for development, the VS Code Ruff Extension and VS Code ty Extension will enable viewing issues from Ruff and ty within your editor. Additionally, Ruff, ty, and CodeQL analysis will be run in the integration workflow.

Orchestration

In production, daily GitHub Actions workflows run to orchestrate the entire data pipeline.

Ver o original no GitHub

Bom para

Relacionados