Flow LLM
styles01/flow-llm
Gateway LLM local para Apple Silicon com monitoramento em tempo real
Ver no GitHubGateway local para LLMs em Apple Silicon que gerencia modelos GGUF e MLX, proxeia requisições OpenAI e Anthropic, e fornece monitoramento em tempo real. Sem necessidade de Ollama ou LM Studio, integra-se perfeitamente com agents como OpenClaw, Hermes e Claude Code.
Instalação
npx convez-ai styles01/flow-llm1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Executar modelos locais GGUF/MLX sem Ollama ou LM Studio
- Integrar agents de código (OpenClaw, Hermes) com modelos locais
- Proxiar APIs OpenAI e Anthropic para modelos locais
- Monitorar em tempo real processamento de tokens e requisições
Veredito

Solução robusta para quem opera modelos locais em Macs com Apple Silicon. O carregamento JIT, monitoramento WebSocket e suporte a agents fazem dele uma escolha técnica sólida, embora exija conhecimento de infraestrutura e setup inicial de backends.
README
A tradução para o português deste README ainda está na fila.
Flow LLM
Local LLM gateway for Apple Silicon
Run GGUF and MLX models locally. Proxy OpenAI & Anthropic API requests. Real-time monitoring. Built for coding agents.
Install · Features · Quick Start · API · Architecture
Flow LLM is a local LLM gateway for macOS. It manages GGUF and MLX models on Apple Silicon, proxies OpenAI- and Anthropic-compatible API requests, and exposes real-time monitoring — so tools like OpenClaw, Hermes, Claude Code, and Codex (via AIRun) can talk to local models without Ollama or LM Studio.

Features
- JIT Model Loading — Models auto-load on first request and auto-unload after idle cooldown. Togglable in Settings (defaults ON, 5 min cooldown). Circuit breaker prevents memory exhaustion
- Real-time Monitor — Per-request lifecycle tracking (queued → prefilling → generating → completed), odometer-style token counter, WebSocket push, idle waveform
- OpenAI & Anthropic APIs — Drop-in proxy for
/v1/chat/completionsand/v1/messages. Streaming and non-streaming, tool calling, system prompts, reasoning/thinking block translation - GGUF & MLX — Run llama.cpp GGUF models or MLX models (text-only and vision) on Apple Silicon with sensible defaults (100K context, flash attention, q4_0 KV cache). Speculative decoding for MLX
- Agent-Ready — Parallel slot support, Anthropic streaming SSE adapter, input token estimation fallback, stuck request pruning
- Connect External — Adopt an already-running llama-server without restarting it. Auto-detects model name
- HuggingFace Browser — Search and download models directly from the UI. Scan local directories for unregistered GGUF files
- Telemetry — TTFT, throughput, token counts per request. Card-based history with color-coded metrics
- Template Validation — Validates chat templates before loading (Jinja syntax, system role, tool calling)
- Single Binary —
pip install -e . && flow. One process, one port (3377). Frontend bundled in the package
Quick Install
curl -fsSL https://raw.githubusercontent.com/styles01/flow-llm/main/setup.sh | bash
Or clone and run:
git clone https://github.com/styles01/flow-llm.git
cd flow-llm && ./setup.sh
flow
Open http://localhost:3377 — API and UI from a single process.
Prerequisites
Flow requires inference backends. Install at least one:
# Required — GGUF models
brew install llama.cpp
# Optional — MLX models (text + vision)
pip install mlx_lm mlx_vlm
Quick Start
1. Start Flow
flow
2. Add a model (one-time setup)
In the UI: Models → search HuggingFace, download and register any model. Or connect a running backend.
Or via API:
curl -X POST http://localhost:3377/api/register-local \
-H "Content-Type: application/json" \
-d '{"gguf_path": "/path/to/model.gguf"}'
3. Point your agent
With JIT (default): Your first inference request auto-loads the model. Nothing else to do.
Without JIT: Turn it off in Settings, then load models explicitly via the Models page or POST /api/models/{id}/load.
{
"models": {
"providers": {
"flow": {
"baseUrl": "http://127.0.0.1:3377/v1",
"apiKey": "flow-local",
"api": "openai-completions"
}
}
}
}
Flow also exposes POST /v1/messages for Claude Code and other Anthropic API tools.
Model Loading Defaults
Flow ships with sensible defaults for Apple Silicon:
| Setting | Default | Why |
|---|---|---|
| Context window | 100,000 tokens | Coding agents need long context |
| Flash attention | On | Critical for long context performance |
| KV cache | q4_0 | 75% memory savings, enables 100K on 48GB |
| GPU layers | -1 (all) | Metal acceleration |
| Parallel slots | 2 | Concurrent agent requests |
| JIT loading | On | Auto-load models on first request |
| JIT cooldown | 300s (5 min) | Auto-unload idle models |
| Auto-update | On | Checks backend versions on startup |
Configurable in Settings page, persisted to ~/.flow/settings.json.
Development
cd server && pip install -e .
cd ../web && npm install && npm run dev
Frontend dev server at http://localhost:5173 proxies API requests to the backend. Rebuild bundled frontend:
cd web && npm run build
Dependencies
Required
| Dependency | Purpose | Install |
|---|---|---|
| Python 3.11+ | Runtime | System |
| llama.cpp | GGUF inference backend | brew install llama.cpp |
| Node.js 18+ | Frontend build | brew install node |
Python packages (installed via pip install -e .)
| Package | Purpose |
|---|---|
| fastapi | Management server and API |
| uvicorn | ASGI server |
| httpx | Async HTTP proxy |
| sqlalchemy | Model registry (SQLite) |
| huggingface-hub | Model search and download |
| jinja2 | Chat template validation |
| psutil | Hardware detection |
| pydantic | Request/response models |
| websockets | Real-time updates |
Optional
| Dependency | Purpose | Install |
|---|---|---|
| mlx_lm / mlx_vlm | MLX inference backends (text + vision) | pip install mlx_lm mlx_vlm |
Connect External Backend
Flow can adopt an already-running backend without restarting it:
curl -X POST http://localhost:3377/api/connect-external \
-H "Content-Type: application/json" \
-d '{"base_url": "http://127.0.0.1:8081"}'
Auto-detects the model name. Unloading kills the backend process and frees memory.
Port Layout
| Port | Service |
|---|---|
| 3377 | Flow management server |
| 5173 | Frontend dev server (Vite) |
| 8081+ | llama.cpp backend processes |
| 8100+ | mlx-openai-server backend processes |
API Endpoints
Management API
| Method | Endpoint | Purpose |
|---|---|---|
| GET | /api/hardware | Hardware info (chip, memory, Metal) |
| GET | /api/models | List all registered models |
| GET | /api/models/{id} | Get model details |
| GET | /api/models/running | List running models |
| POST | /api/models/{id}/load | Load a model |
| POST | /api/models/{id}/unload | Unload a model |
| DELETE | /api/models/{id} | Delete a model |
| POST | /api/models/download | Download from HuggingFace |
| POST | /api/models/scan | Scan for unregistered GGUF files |
| POST | /api/register-local | Register a local GGUF file |
| POST | /api/connect-external | Connect to a running backend |
| GET | /api/settings | Get default loading settings |
| PUT | /api/settings | Update settings |
| GET | /api/downloads | Download progress |
| GET | /api/hf/search?q= | Search HuggingFace |
| GET | /api/telemetry | Request telemetry records |
| GET | /api/requests | Active request tracker |
| POST | /api/requests/clear-stuck | Clear stuck requests |
| GET | /api/logs | Backend logs |
| GET | /api/model-activity | Per-slot activity and metrics |
| GET | /api/health | Health check |
OpenAI-Compatible Proxy
| Method | Endpoint | Purpose |
|---|---|---|
| POST | /v1/chat/completions | Chat completions (streaming + non-streaming) |
| POST | /v1/messages | Anthropic Messages API |
| GET | /v1/models | List available models |
WebSocket
| Endpoint | Purpose |
|---|---|
/ws | Real-time updates (request lifecycle, slot state, metrics, model events) |
Changelog
v1.5.0 — JIT model loading
JIT (Just-In-Time) model loading is the headline feature — models auto-load when an inference request arrives, then auto-unload after a configurable idle cooldown. It's on by default but fully optional (turn it off in Settings for explicit control). Circuit breaker prevents memory exhaustion by estimating requirements and evicting idle models oldest-first. Cooldown tasks check for active in-flight requests before unloading, so streaming responses are never interrupted. Works with both GGUF and MLX backends.
Also in this release:
- Speculative decoding for MLX — Draft model path and num draft tokens fields in the Load dialog, forwarded to mlx-openai-server for 2-3x throughput on supported models.
- Anthropic thinking blocks —
reasoning_contentfrom OpenAI backends is now translated to Anthropicthinkingcontent blocks in both streaming SSE and non-streaming responses. - Monitor polling fix — Polling fallback no longer overwrites fresher WebSocket-pushed request state, preventing stage regression.
- ProcessManager thread-safety —
asyncio.Lockaround all mutations, preventing races during compound operations.
v1.1.0 — Qwen 3.6 MLX tool calling + warmup UX
Qwen 3.6 MLX support (unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit)
Getting Qwen 3.6 working reliably with Hermes-style agents (38 tools, 100+ message sessions, streaming) required fixing a cascade of interacting bugs:
- XML tool call format — this unsloth quantization generates
<function=name><parameter=x>v</parameter></function>inside<tool_call>tags (Qwen3-Coder format), not Hermes JSON. The proxy now parses both formats and normalises totool_calls[]. - HTML entity cascade — when a tool call leaked as text, Telegram HTML-escaped it (
<tool_call>→<tool_call>) in session storage. The model then mimicked the escaped format on every subsequent turn, snowballing until the session was unrecoverable. Fixed withhtml.unescape()as the first step in rescue. - Truncated responses — Hermes sends
max_tokens=4096; with the reasoning parser active, thinking tokens consumed the budget leaving 2–5 tokens for the actual reply. The proxy now dropsmax_tokenswhen a reasoning parser is active, letting the full context window (262K) be the cap. </think>bleeding into content — the reasoning parser only activates when it sees a<think>opening tag in the stream, but Qwen3's generation prefix omits it. Any thinking content was flowing intocontent. Fixed with a stripping pass that moves…\n</think>\ntoreasoning_content.- Half-warm model poisoning — requests arriving during weight loading returned malformed responses that agents stored permanently in session history. The proxy now returns 503 until the backend health check passes.
Preset and template handling
- Built-in "Qwen3.6 — Tools (stable)" preset covers all required load params: 262K context,
qwen3reasoning/tool parsers, Hermes JSON chat template, trust-remote-code. - Qwen chat template auto-fill now runs regardless of whether
tool_call_parserwas explicitly provided (was previously skipped).
Warmup UX improvements
- Monitor page shows a real loading percentage bar (
Loading weights: 42%) parsed from mlx-lm's stderr during weight loading, so you know exactly where the model is instead of just an amber "warming up" badge. - Backend-ready guard: the proxy rejects requests with 503 while weights are still loading — agents get a clean retryable error instead of a malformed response that corrupts their context.
v1.0.0 — Real-time Monitor + request lifecycle tracking
- Per-request lifecycle tracking: queued → prefilling → generating → sending → completed
- WebSocket push for real-time monitor updates
- LM Studio-style odometer token counter
- PWA manifest, app icons, theme-color meta tag
- Telemetry page redesigned with card layout and color-coded TTFT