convez.ai
AgenteDevOps & InfraestruturaIntermediário
styles01

Flow LLM

styles01/flow-llm

Gateway LLM local para Apple Silicon com monitoramento em tempo real

Ver no GitHub
InicialPluginScore · de 0 a 100

Gateway local para LLMs em Apple Silicon que gerencia modelos GGUF e MLX, proxeia requisições OpenAI e Anthropic, e fornece monitoramento em tempo real. Sem necessidade de Ollama ou LM Studio, integra-se perfeitamente com agents como OpenClaw, Hermes e Claude Code.

Instalação

$npx convez-ai styles01/flow-llm

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
2
Forks
0
Acompanhando
1
Saúde
42%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
2

Casos de uso

  • Executar modelos locais GGUF/MLX sem Ollama ou LM Studio
  • Integrar agents de código (OpenClaw, Hermes) com modelos locais
  • Proxiar APIs OpenAI e Anthropic para modelos locais
  • Monitorar em tempo real processamento de tokens e requisições

Veredito

Solução robusta para quem opera modelos locais em Macs com Apple Silicon. O carregamento JIT, monitoramento WebSocket e suporte a agents fazem dele uma escolha técnica sólida, embora exija conhecimento de infraestrutura e setup inicial de backends.

README

A tradução para o português deste README ainda está na fila.

Flow LLM

Local LLM gateway for Apple Silicon

Run GGUF and MLX models locally. Proxy OpenAI & Anthropic API requests. Real-time monitoring. Built for coding agents.

License: MIT Python 3.11+ Platform: Apple Silicon Buy me a coffee

Install · Features · Quick Start · API · Architecture


Flow LLM is a local LLM gateway for macOS. It manages GGUF and MLX models on Apple Silicon, proxies OpenAI- and Anthropic-compatible API requests, and exposes real-time monitoring — so tools like OpenClaw, Hermes, Claude Code, and Codex (via AIRun) can talk to local models without Ollama or LM Studio.

Flow LLM Monitor

Features

  • JIT Model Loading — Models auto-load on first request and auto-unload after idle cooldown. Togglable in Settings (defaults ON, 5 min cooldown). Circuit breaker prevents memory exhaustion
  • Real-time Monitor — Per-request lifecycle tracking (queued → prefilling → generating → completed), odometer-style token counter, WebSocket push, idle waveform
  • OpenAI & Anthropic APIs — Drop-in proxy for /v1/chat/completions and /v1/messages. Streaming and non-streaming, tool calling, system prompts, reasoning/thinking block translation
  • GGUF & MLX — Run llama.cpp GGUF models or MLX models (text-only and vision) on Apple Silicon with sensible defaults (100K context, flash attention, q4_0 KV cache). Speculative decoding for MLX
  • Agent-Ready — Parallel slot support, Anthropic streaming SSE adapter, input token estimation fallback, stuck request pruning
  • Connect External — Adopt an already-running llama-server without restarting it. Auto-detects model name
  • HuggingFace Browser — Search and download models directly from the UI. Scan local directories for unregistered GGUF files
  • Telemetry — TTFT, throughput, token counts per request. Card-based history with color-coded metrics
  • Template Validation — Validates chat templates before loading (Jinja syntax, system role, tool calling)
  • Single Binary — pip install -e . && flow. One process, one port (3377). Frontend bundled in the package

Quick Install

curl -fsSL https://raw.githubusercontent.com/styles01/flow-llm/main/setup.sh | bash

Or clone and run:

git clone https://github.com/styles01/flow-llm.git
cd flow-llm && ./setup.sh
flow

Open http://localhost:3377 — API and UI from a single process.

Prerequisites

Flow requires inference backends. Install at least one:

# Required — GGUF models
brew install llama.cpp

# Optional — MLX models (text + vision)
pip install mlx_lm mlx_vlm

Quick Start

1. Start Flow

flow

2. Add a model (one-time setup)

In the UI: Models → search HuggingFace, download and register any model. Or connect a running backend.

Or via API:

curl -X POST http://localhost:3377/api/register-local \
  -H "Content-Type: application/json" \
  -d '{"gguf_path": "/path/to/model.gguf"}'

3. Point your agent

With JIT (default): Your first inference request auto-loads the model. Nothing else to do.

Without JIT: Turn it off in Settings, then load models explicitly via the Models page or POST /api/models/{id}/load.

{
  "models": {
    "providers": {
      "flow": {
        "baseUrl": "http://127.0.0.1:3377/v1",
        "apiKey": "flow-local",
        "api": "openai-completions"
      }
    }
  }
}

Flow also exposes POST /v1/messages for Claude Code and other Anthropic API tools.

Model Loading Defaults

Flow ships with sensible defaults for Apple Silicon:

SettingDefaultWhy
Context window100,000 tokensCoding agents need long context
Flash attentionOnCritical for long context performance
KV cacheq4_075% memory savings, enables 100K on 48GB
GPU layers-1 (all)Metal acceleration
Parallel slots2Concurrent agent requests
JIT loadingOnAuto-load models on first request
JIT cooldown300s (5 min)Auto-unload idle models
Auto-updateOnChecks backend versions on startup

Configurable in Settings page, persisted to ~/.flow/settings.json.

Development

cd server && pip install -e .
cd ../web && npm install && npm run dev

Frontend dev server at http://localhost:5173 proxies API requests to the backend. Rebuild bundled frontend:

cd web && npm run build

Dependencies

Required

DependencyPurposeInstall
Python 3.11+RuntimeSystem
llama.cppGGUF inference backendbrew install llama.cpp
Node.js 18+Frontend buildbrew install node

Python packages (installed via pip install -e .)

PackagePurpose
fastapiManagement server and API
uvicornASGI server
httpxAsync HTTP proxy
sqlalchemyModel registry (SQLite)
huggingface-hubModel search and download
jinja2Chat template validation
psutilHardware detection
pydanticRequest/response models
websocketsReal-time updates

Optional

DependencyPurposeInstall
mlx_lm / mlx_vlmMLX inference backends (text + vision)pip install mlx_lm mlx_vlm

Connect External Backend

Flow can adopt an already-running backend without restarting it:

curl -X POST http://localhost:3377/api/connect-external \
  -H "Content-Type: application/json" \
  -d '{"base_url": "http://127.0.0.1:8081"}'

Auto-detects the model name. Unloading kills the backend process and frees memory.

Port Layout

PortService
3377Flow management server
5173Frontend dev server (Vite)
8081+llama.cpp backend processes
8100+mlx-openai-server backend processes

API Endpoints

Management API

MethodEndpointPurpose
GET/api/hardwareHardware info (chip, memory, Metal)
GET/api/modelsList all registered models
GET/api/models/{id}Get model details
GET/api/models/runningList running models
POST/api/models/{id}/loadLoad a model
POST/api/models/{id}/unloadUnload a model
DELETE/api/models/{id}Delete a model
POST/api/models/downloadDownload from HuggingFace
POST/api/models/scanScan for unregistered GGUF files
POST/api/register-localRegister a local GGUF file
POST/api/connect-externalConnect to a running backend
GET/api/settingsGet default loading settings
PUT/api/settingsUpdate settings
GET/api/downloadsDownload progress
GET/api/hf/search?q=Search HuggingFace
GET/api/telemetryRequest telemetry records
GET/api/requestsActive request tracker
POST/api/requests/clear-stuckClear stuck requests
GET/api/logsBackend logs
GET/api/model-activityPer-slot activity and metrics
GET/api/healthHealth check

OpenAI-Compatible Proxy

MethodEndpointPurpose
POST/v1/chat/completionsChat completions (streaming + non-streaming)
POST/v1/messagesAnthropic Messages API
GET/v1/modelsList available models

WebSocket

EndpointPurpose
/wsReal-time updates (request lifecycle, slot state, metrics, model events)

Changelog

v1.5.0 — JIT model loading

JIT (Just-In-Time) model loading is the headline feature — models auto-load when an inference request arrives, then auto-unload after a configurable idle cooldown. It's on by default but fully optional (turn it off in Settings for explicit control). Circuit breaker prevents memory exhaustion by estimating requirements and evicting idle models oldest-first. Cooldown tasks check for active in-flight requests before unloading, so streaming responses are never interrupted. Works with both GGUF and MLX backends.

Also in this release:

  • Speculative decoding for MLX — Draft model path and num draft tokens fields in the Load dialog, forwarded to mlx-openai-server for 2-3x throughput on supported models.
  • Anthropic thinking blocks — reasoning_content from OpenAI backends is now translated to Anthropic thinking content blocks in both streaming SSE and non-streaming responses.
  • Monitor polling fix — Polling fallback no longer overwrites fresher WebSocket-pushed request state, preventing stage regression.
  • ProcessManager thread-safety — asyncio.Lock around all mutations, preventing races during compound operations.

v1.1.0 — Qwen 3.6 MLX tool calling + warmup UX

Qwen 3.6 MLX support (unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit)

Getting Qwen 3.6 working reliably with Hermes-style agents (38 tools, 100+ message sessions, streaming) required fixing a cascade of interacting bugs:

  • XML tool call format — this unsloth quantization generates <function=name><parameter=x>v</parameter></function> inside <tool_call> tags (Qwen3-Coder format), not Hermes JSON. The proxy now parses both formats and normalises to tool_calls[].
  • HTML entity cascade — when a tool call leaked as text, Telegram HTML-escaped it (<tool_call> → &lt;tool_call&gt;) in session storage. The model then mimicked the escaped format on every subsequent turn, snowballing until the session was unrecoverable. Fixed with html.unescape() as the first step in rescue.
  • Truncated responses — Hermes sends max_tokens=4096; with the reasoning parser active, thinking tokens consumed the budget leaving 2–5 tokens for the actual reply. The proxy now drops max_tokens when a reasoning parser is active, letting the full context window (262K) be the cap.
  • </think> bleeding into content — the reasoning parser only activates when it sees a <think> opening tag in the stream, but Qwen3's generation prefix omits it. Any thinking content was flowing into content. Fixed with a stripping pass that moves …\n</think>\n to reasoning_content.
  • Half-warm model poisoning — requests arriving during weight loading returned malformed responses that agents stored permanently in session history. The proxy now returns 503 until the backend health check passes.

Preset and template handling

  • Built-in "Qwen3.6 — Tools (stable)" preset covers all required load params: 262K context, qwen3 reasoning/tool parsers, Hermes JSON chat template, trust-remote-code.
  • Qwen chat template auto-fill now runs regardless of whether tool_call_parser was explicitly provided (was previously skipped).

Warmup UX improvements

  • Monitor page shows a real loading percentage bar (Loading weights: 42%) parsed from mlx-lm's stderr during weight loading, so you know exactly where the model is instead of just an amber "warming up" badge.
  • Backend-ready guard: the proxy rejects requests with 503 while weights are still loading — agents get a clean retryable error instead of a malformed response that corrupts their context.

v1.0.0 — Real-time Monitor + request lifecycle tracking

  • Per-request lifecycle tracking: queued → prefilling → generating → sending → completed
  • WebSocket push for real-time monitor updates
  • LM Studio-style odometer token counter
  • PWA manifest, app icons, theme-color meta tag
  • Telemetry page redesigned with card layout and color-coded TTFT

License

MIT

Ver o original no GitHub

Bom para

Relacionados