AcornOps LLM Gateway
acornops/llm-gateway
Gateway unificado para LLMs e ferramentas MCP com segurança em produção
Ver no GitHubGateway LLM de nível produção que centraliza acesso a múltiplos provedores de modelos (OpenAI, Anthropic, Gemini) e gerencia ferramentas via MCP (Model Context Protocol). Oferece autenticação JWT, isolamento multi-tenant, brokering de ferramentas e observabilidade completa com logs estruturados e métricas Prometheus.
Instalação
npx convez-ai acornops/llm-gateway1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Centralizar chamadas a múltiplos provedores LLM com fallback e load balancing
- Executar ferramentas MCP de forma segura com isolamento de secrets por tenant
- Implementar políticas de segurança (JWT, JWKS) e auditoria em camada gateway
- Distribuir requisições com streaming NDJSON e observabilidade completa
Veredito

Solução robusta para orquestração de LLMs e MCP em arquiteturas multi-tenant e produção. Exige expertise DevOps/Backend, mas fornece abstrações bem-definidas (contracts), segurança em camadas e observabilidade enterprise-grade.
README
A tradução para o português deste README ainda está na fila.
Status
This repository owns the gateway service code, production image, health/readiness contract, metrics, and service-level docs. Full-system deployment wiring belongs in acornops-deployment.
Agent-Assisted Development
This repository supports human and agent-assisted development. Start coding agents from this repository root for llm-gateway-only work, and from the acornops-workspace root for changes that touch multiple AcornOps repositories.
Contracts
Cross-repo contract documentation lives in docs/contracts/README.md. Treat that file as the source of truth for control-plane, execution-engine, and MCP integration boundaries.
Machine-readable contract data lives in docs/contracts/manifest.json.
Run task contracts:check to mechanically verify the documented contracts against the implementation.
Coverage is generated in CI with pytest-cov, uploaded as a workflow artifact, and published to Codecov when CODECOV_TOKEN is configured for the repository.
Documentation
Primary docs:
AGENTS.mdARCHITECTURE.mddocs/index.mddocs/DEVELOPMENT.mddocs/OPERATIONS.md- Whole-system architecture:
../docs/system-architecture.md
Features
- LLM Proxy: Unified interface for OpenAI, Anthropic, and Gemini (Google).
- Streaming Support: NDJSON-based streaming for provider generations.
- MCP Tool Broker: Execute tools via Model Context Protocol (MCP) servers.
- Security:
- JWT authentication with JWKS validation.
- Multi-tenant isolation for secrets and tools.
- Pluggable secret storage: encrypted Postgres (AES-256-GCM) or HashiCorp Vault (KV v2).
- Observability: Structured JSON logging and Prometheus metrics.
- Resilience: Configurable timeouts, bounded retries for idempotent outbound dependency calls, and circuit breakers for repeated provider, MCP, and Vault failures.
Architecture
flowchart LR
EE[Execution Engine] -->|Bearer JWT + JSON| GW["llm-gateway"]
GW --> AUTH[Auth & Policy]
AUTH --> LLM[LLM Proxy Module]
AUTH --> MCP[MCP Tool Broker]
LLM --> OPENAI[OpenAI]
LLM --> ANTH[Anthropic]
LLM --> GEMINI[Gemini]
MCP --> REG[Tool Registry]
MCP --> MOCK[Mock MCP Server]
GW --> OBS[Logs & Metrics]
GW --> SEC[Secret Store]
Setup & Development
Prerequisites
- Python 3.12.11. Local development, CI, and production images are standardized on the same Python patch release.
- Docker & Docker Compose
- Postgres (if running locally without Docker)
- Redis (if running locally without Docker)
The repository pins .python-version to 3.12.11 for pyenv-compatible local workflows. Recreate the virtualenv after changing Python versions:
pyenv install 3.12.11
pyenv local 3.12.11
python -m venv .venv
.venv/bin/python -m pip install --upgrade pip
.venv/bin/python -m pip install '.[test]' -c constraints.txt
task validate
Compose Layout
docker-compose.yml: base/default component runtime (gateway,postgres,redis, migration job). It is useful for local integration and image smoke tests; full deployment topology lives inacornops-deployment.docker-compose.override.yml: local development additions (local builds, host ports,mock-auth,mock-mcp, seed data).- base compose defaults
AUTH_JWKS_URLtohttps://api.<BASE_DOMAIN>/api/v1/auth/jwks.json(BASE_DOMAIN=acornops.devby default), gates the gateway on/ready, and usesLLM_GATEWAY_IMAGEfor explicit image selection.
Secret Backends
- Default backend:
SECRETS_BACKEND=database. - Optional Vault backend: set
SECRETS_BACKEND=vaultand provide:VAULT_ADDRVAULT_TOKEN- optional
VAULT_NAMESPACE - optional
VAULT_MOUNT(defaultsecret) - optional
VAULT_PATH_PREFIX(defaultacornops) - optional
VAULT_TIMEOUT_MS/VAULT_VERIFY_TLS
- LLM provider credentials are workspace-scoped and are configured through workspace AI settings.
- MCP auth secrets remain target-scoped by
workspace_id,target_id, andtarget_type. - For local Vault testing, the override includes a
vaultservice under thevaultprofile:
docker compose --profile vault up -d --build
Internal MCP Admin API
The control-plane manages target-scoped MCP and built-in tool configuration through llm-gateway internal endpoints. Kubernetes callers pass target_type: "kubernetes":
GET /api/v1/internal/mcp/serversGET /api/v1/internal/mcp/toolsPATCH /api/v1/internal/mcp/tools/{tool_name}POST /api/v1/internal/mcp/serversPATCH /api/v1/internal/mcp/servers/{server_id}POST /api/v1/internal/mcp/servers/{server_id}/testDELETE /api/v1/internal/mcp/servers/{server_id}
All of the above require Authorization: Bearer <ADMIN_API_TOKEN> and explicit workspace_id + target_id + target_type query/body scope where applicable.
MCP Server Discovery Contract
For remote MCP servers managed by the management console's MCP Servers tab:
- tool discovery is expected via MCP
tools/list. - tool execution is expected via MCP
tools/call. - this gateway supports the following discovery transports in order:
POST {server_url}/tools/listGET {server_url}/tools/list- JSON-RPC fallback:
POST {server_url}with methodtools/list
- and supports tool calls via:
POST {server_url}/tools/call- JSON-RPC fallback:
POST {server_url}with methodtools/call
On server create (when no explicit tool list is supplied), llm-gateway discovers
tools, stores them disabled, and sanitizes remote descriptions/schemas before
they are shown for admin review. A discovered external tool is not returned to
runtime tool lists or sent to an LLM until an admin explicitly enables it with a
reviewed capability.
Each MCP server also tracks connection_status, last_discovery_at, and last_discovery_error so UI can surface discovery health.
If discovery fails or returns empty, server creation still succeeds but no tools are mapped until discovery succeeds.
MCP server public_headers are visible non-secret metadata. Credential-bearing headers must use secret-backed auth fields.
MCP egress is protected by default. Remote MCP URLs must be absolute HTTP(S) URLs without embedded credentials; production requires HTTPS and rejects DNS results in loopback, link-local, multicast, private, reserved, or unspecified address ranges. Local development allows Docker service-name targets such as mock-mcp. Private production MCP targets require an explicit MCP_EGRESS_ALLOWED_HOSTS allowlist entry or MCP_EGRESS_ALLOW_PRIVATE_NETWORKS=true.
LLM and tool-call limits are configured by LLM_RATE_LIMIT_PER_WINDOW, TOOL_RATE_LIMIT_PER_WINDOW, and RATE_LIMIT_WINDOW_SECONDS. In production, REQUIRE_REDIS_RATE_LIMITS_IN_PRODUCTION=true makes missing Redis configuration a startup error.
Runtime JWKS checks are configured with JWKS_CACHE_TTL_SECONDS,
JWKS_READINESS_MAX_STALENESS_SECONDS, and REQUIRE_JWKS_READINESS. Production
defaults require JWKS readiness so gateway instances only receive traffic when
they can validate control-plane-issued run tokens.
Tool registry and secret values are cached in-process. When Redis is configured,
tool registry and secret writes publish cache invalidation events for other
gateway instances. Without Redis, propagation delay is bounded by
TOOL_REGISTRY_CACHE_TTL_SEC and SECRETS_CACHE_TTL_SEC. Production requires
SECRETS_CACHE_TTL_SEC=0 so plaintext provider secrets are not retained in the
gateway process cache.
Run Modes
- Component-only local development (recommended in this repo):
docker compose up -d --build
This starts:
- gateway (
http://localhost:8001) - postgres (
localhost:5432) - redis (
localhost:6379) - mock-auth (
http://localhost:8003) - mock-mcp (
http://localhost:8002)
The gateway and mock auth/MCP services run with Uvicorn --reload, so code changes are reflected immediately.
- Component-only image smoke test:
docker compose -f docker-compose.yml up -d
Use LLM_GATEWAY_IMAGE=ghcr.io/acornops/llm-gateway:<release> to select a release image. This compose file does not replace the full AcornOps deployment manifests.
- Full AcornOps stack (all components together):
cd ../acornops-deployment
task local-up
This full-stack flow uses the deployment repo Taskfile.yml and requires the task CLI to be installed.
Use full-stack mode when you want to validate control-plane-issued auth and real end-to-end traffic.
Do not run this repository's local compose stack and acornops-deployment local stack at the same time on the same host ports.
If dependencies change (requirements.txt or requirements.lock), rebuild once:
docker compose up -d --build
Local Development (Docker Required)
Docker is required for local development and testing to ensure consistency with the production environment.
Optional re-seed command:
docker compose run --rm gateway-init sh -c "alembic upgrade head && python scripts/seed_db.py"
To test real inference in local development, set dev seed provider API keys before seeding. The seed job stores these as workspace-scoped provider credentials; production traffic should configure credentials through workspace settings. Gemini is the recommended demo default:
export ACORNOPS_DEV_SEED_GEMINI_API_KEY='<your-gemini-api-key>'
docker compose run --rm gateway-init sh -c "alembic upgrade head && python scripts/seed_db.py"
The Gemini adapter uses Google's current google-genai SDK.
Image Consumption
The gateway publishes non-root container images for the broader AcornOps
deployment. In production, consume those images through acornops-deployment
and pin a release tag instead of using a mutable tag. Release builds publish
SBOM/provenance metadata through GitHub Actions. The gateway image installs from
the hash-locked requirements.lock file.
1. Pull the Image (Optional)
Images are built and pushed to GHCR by the release workflow.
docker pull ghcr.io/acornops/llm-gateway:<release-tag>
2. Local Integration Example
For local integration testing outside the full AcornOps deployment, use an explicit release tag in compose:
services:
gateway:
image: ghcr.io/acornops/llm-gateway:<release-tag>
ports:
- "8001:8001"
environment:
- DATABASE_URL=postgresql+asyncpg://gateway_user:gateway_password@postgres:5432/gateway
- REDIS_URL=redis://redis:6379/0
- SECRETS_KEK_BASE64=SglIGBscu1EgQ+AlpqJLADNN9QmCzS9d1ZvK3oT/e5s=
- AUTH_JWKS_URL=http://mock-auth:8003/jwks.json
- AUTH_ISSUER=llm-gateway
- AUTH_AUDIENCE=execution-gateway
depends_on:
- postgres
- redis
postgres:
image: postgres:16
environment:
- POSTGRES_USER=gateway_user
- POSTGRES_PASSWORD=gateway_password
- POSTGRES_DB=gateway
redis:
image: redis:7-alpine
mock-auth:
build:
context: ../llm-gateway
dockerfile: deployments/Dockerfile.mock-auth
ports:
- "8003:8003"
3. Point a Local Client to the Gateway
Configure a local client to use the gateway's address:
client = GatewayLlmClient(url="http://gateway:8001", token="your-jwt-token")
4. Database Setup in Integration
Remember to run the migrations and seed the database in your integration environment, or mount a pre-seeded volume to Postgres.
Testing
Running Tests
# Run all tests
pytest
# Run with coverage
pytest --cov=app
Manual Testing
-
Get a Mock Token: The
mock-authservice runs on port 8003:curl -X POST http://localhost:8003/token \ -H "Content-Type: application/json" \ -d '{ "run_id": "5e709a9c-2481-4baa-aec2-ca193c50167d", "workspace_id": "4b930d98-add9-4924-ab26-3c16d96ec373", "target_id": "5b006e4c-509c-458a-9f02-5aafbdc01ade", "target_type": "kubernetes", "session_id": "6e30d188-7e4f-4cce-a368-40b34004d725" }' -
Stream Provider Generations:
curl -X POST http://localhost:8001/api/v1/llm/generations:stream \ -H "Authorization: Bearer <TOKEN>" \ -H "Content-Type: application/json" \ -d '{ "run_id": "5e709a9c-2481-4baa-aec2-ca193c50167d", "workspace_id": "4b930d98-add9-4924-ab26-3c16d96ec373", "target_id": "5b006e4c-509c-458a-9f02-5aafbdc01ade", "target_type": "kubernetes", "session_id": "6e30d188-7e4f-4cce-a368-40b34004d725", "provider": "gemini", "model": "gemini-2.0-flash", "messages": [{"role": "user", "content": "Hello!"}] }' -
Call a Tool:
curl -X POST http://localhost:8001/api/v1/mcp/tool-call \ -H "Authorization: Bearer <TOKEN>" \ -H "Content-Type: application/json" \ -d '{ "run_id": "5e709a9c-2481-4baa-aec2-ca193c50167d", "workspace_id": "4b930d98-add9-4924-ab26-3c16d96ec373", "target_id": "5b006e4c-509c-458a-9f02-5aafbdc01ade", "target_type": "kubernetes", "tool": "get_weather", "arguments": {"location": "San Francisco"} }'
API Documentation
Once the server is running, visit:
- Swagger UI:
http://localhost:8001/docs - ReDoc:
http://localhost:8001/redoc - OpenAPI JSON:
http://localhost:8001/openapi.json
ENABLE_API_DOCS controls docs exposure. Local override sets it to true; base/production-style compose defaults it to false.
Observability
- Metrics:
http://localhost:8001/metrics(Prometheus format) - Logs: JSON formatted logs emitted to stdout.
- Liveness:
http://localhost:8001/healthreturns process health only. - Readiness:
http://localhost:8001/readyverifies database connectivity, Redis connectivity when Redis-backed features are enabled, JWKS reachability and cache freshness, and secret-backend health. It returns503with per-dependency details when the gateway should not receive traffic.
Validation
Run the checks that match the change:
task python:checktask contracts:checktask harness:checktask linttask validatetask unit-testin a provisioned environment when auth, provider, or MCP behavior changes