convez.ai
BomPluginScore · de 0 a 100

Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across SGLang, vLLM, TRT-LLM, OpenAI, Gemini & more. Industry-first gRPC pipeline, KV cache-aware routing, chat history, tokenization caching, Responses API, embeddings, WASM plugins, MCP, and multi-tenant auth.

Instalação

$npx convez-ai lightseekorg/smg

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
731
Forks
226
Acompanhando
2
Saúde
75%
Último commit
há 1 dia
Versão
v1.15.0
Issues abertas
193
Contribuidores
104

README

A tradução para o português deste README ainda está na fila.

High-performance model-routing gateway for large-scale LLM deployments. Centralizes worker lifecycle management, balances traffic across HTTP/gRPC/OpenAI-compatible backends, and provides enterprise-ready control over history storage, MCP tooling, and privacy-sensitive workflows.

Why SMG?

🚀 Maximize GPU UtilizationCache-aware routing understands your inference engine's KV cache state—whether SGLang, vLLM, or TensorRT-LLM—to reuse prefixes and reduce redundant computation.
🔌 One API, Any BackendRoute to self-hosted models (SGLang, vLLM, TensorRT-LLM) or cloud providers (OpenAI, Anthropic, Gemini, Bedrock, and more) through a single unified endpoint.
⚡ Built for SpeedNative Rust with gRPC pipelines, sub-millisecond routing decisions, and zero-copy tokenization. Circuit breakers and automatic failover keep things running.
🔒 Enterprise ControlMulti-tenant rate limiting with OIDC, WebAssembly plugins for custom logic, and a privacy boundary that keeps conversation history within your infrastructure.
📊 Full Observability40+ Prometheus metrics, OpenTelemetry tracing, and structured JSON logs with request correlation—know exactly what's happening at every layer.

API Coverage: OpenAI Chat/Completions/Embeddings, Responses API for agents, Anthropic Messages, and MCP tool execution.

Quick Start

Install — pick your preferred method:

# Docker
docker pull lightseekorg/smg:latest

# Python
pip install smg

# Rust
cargo install smg

Run — point SMG at your inference workers:

# Single worker
smg launch --worker-urls http://localhost:8000

# Multiple workers with cache-aware routing
smg launch --worker-urls http://gpu1:8000 http://gpu2:8000 --policy cache_aware

# With high availability mesh
smg launch --worker-urls http://gpu1:8000 --enable-mesh \
  --mesh-advertise-host 10.0.0.1 --mesh-peer-urls 10.0.0.2:39527

Use — send requests to the gateway:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3", "messages": [{"role": "user", "content": "Hello!"}]}'

That's it. SMG is now load-balancing requests across your workers.

Supported Backends

Self-HostedCloud Providers
vLLMOpenAI
SGLangAnthropic
TensorRT-LLMGoogle Gemini
OllamaAWS Bedrock
Any OpenAI-compatible serverAzure OpenAI

Features

FeatureDescription
8 Routing Policiescache_aware, round_robin, power_of_two, consistent_hashing, prefix_hash, manual, random, bucket
gRPC PipelineNative gRPC with streaming, reasoning extraction, and tool call parsing
MCP IntegrationConnect external tool servers via Model Context Protocol
High AvailabilityMesh networking with SWIM protocol for multi-node deployments
Chat HistoryPluggable storage: PostgreSQL, Oracle, Redis, or in-memory
WASM PluginsExtend with custom WebAssembly logic
ResilienceCircuit breakers, retries with backoff, rate limiting

Documentation

Getting StartedInstallation and first steps
ArchitectureHow SMG works
ConfigurationCLI reference and options
API ReferenceOpenAI-compatible endpoints
Kubernetes SetupIn-cluster discovery and production setup

Contributing

We welcome contributions! See Contributing Guide for details.

Ver o original no GitHub

Bom para

Relacionados