ChatWalaʻau - Runtime de Agentes IA Local
motojinc25/chatwalaau
Chat UI + Agentes + RAG + MCP em um comando pip
Ver no GitHubRuntime full-stack para agentes IA executado localmente via pip install. Integra interface de chat moderna, agentes, ferramentas customizadas, pipeline RAG, suporte MCP nativo e API compatível com OpenAI — tudo sem cloud ou Docker.
Instalação
npx convez-ai motojinc25/chatwalaau1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Desenvolver e testar agentes IA localmente sem dependências cloud
- Integrar RAG e ferramentas customizadas em aplicações
- Expor agentes como API OpenAI-compatível para terceiros
- Análise multimodal (texto, imagem, busca web) com histórico persistente
Veredito

Solução sólida para quem precisa de runtime IA standalone com UI pronta. A arquitetura localhost e suporte MCP nativo eliminam fricções comuns. Recomendado para equipes que precisam controlar dados localmente e expor agentes via API padronizada.
README
A tradução para o português deste README ainda está na fila.
ChatWalaʻau
The localhost AI Agent Runtime -- Chat UI, Tools, RAG, and MCP in one pip install
ChatWalaʻau is a full-stack AI agent runtime that runs entirely on localhost. It connects a modern chat UI to AI agents via the AG-UI protocol, with built-in tools, RAG pipeline, MCP integration, and an OpenAI-compatible API -- all from a single pip install.
About the Name
"Walaʻau" (wah-la-OW) is a Hawaiian word meaning "to chat, talk, or converse." We chose it because it captures what the agent does, in the language of the place where the project is built.
Hawaiian (ʻōlelo Hawaiʻi) is an indigenous language of the Hawaiian Islands. After a long period of suppression, it is now in active revitalization. We use this word with respect and gratitude.
Why ChatWalaʻau?
- One command, full stack --
pip install chatwalaaugives you Chat UI + Agent Runtime + Tools + RAG. No Docker, no cloud setup. - MCP native -- Zero-config first run via bundled defaults. Connect any MCP server. MCP Apps render interactive UI in chat.
- Your data stays local -- File-based sessions, ChromaDB vectors, and uploads never leave your machine.
- OpenAI-compatible API -- Expose your agent as
/v1/responsesfor any app using the OpenAI SDK.
Hawaii-built, powered by Microsoft Agent Framework
UI Preview
Quick Start
pip install chatwalaau
chatwalaau init
# Edit .env and set AZURE_OPENAI_ENDPOINT
az login
chatwalaau
Open: http://localhost:8000/chat
On a corporate network (TLS-intercepting proxy / in-house root CA)? Install with the
corpextras so Python honours your OS certificate store instead of the bundledcertifi:pip install "chatwalaau[corp]"Adds
pip-system-certs. Default installs are unaffected.
Features
Chat & UI
- Chat with AI agents via AG-UI protocol (SSE streaming)
- Rich message rendering: Markdown, code blocks, math (KaTeX), Mermaid diagrams
- LLM reasoning visualization with collapsible thinking blocks
- Web search with inline citation links (OpenAI hosted web search for OpenAI-family models; Anthropic hosted web search for Claude models)
- Voice input via microphone with Whisper transcription (supports
whisper,gpt-4o-transcribe, andgpt-realtime-whispervia Realtime API) - Text-to-Speech playback and download, with a selectable provider: ElevenLabs or Azure OpenAI Realtime voice models (e.g.
gpt-realtime-2) - Multimodal image analysis (file attachment, drag-and-drop, URL)
- Session management: save, search, pin, archive, fork, rename
- Conversation navigator: a floating right-edge rail of your questions in long chats; click to open a shortcut list and jump straight to any of your messages (shown on the full-page chat when the window is wide enough)
- Context window consumption display with warning levels
- Per-turn token usage display
- Three layout scenarios: Chat, Popup, Sidebar
- Multilingual chat with browser auto-translation suppressed
- Smooth large sessions: opening a long conversation and continuing to chat stays responsive -- only the streaming message re-renders, so scrolling and typing no longer slow down as the message count grows
Agent Tools
- Image generation, editing, and Canvas mask editor via Azure OpenAI gpt-image-1.5
- Weather tools with rich card widgets (Open-Meteo, no API key)
- Coding tools (file read/write, shell execution, file search)
- Tool approval workflow: destructive coding tools (
bash_execute,file_write) pause for an inline Approve / Reject decision by default; one env var (TOOL_APPROVAL_MODE=skip) restores autonomous execution - Prompt Templates: save, manage, and insert reusable prompts from "+" menu and message actions
- Agent Skills: portable domain knowledge packages with progressive disclosure
Platform
- Global Agent Identity: a single
.agent/IDENTITY.mdfile defines the agent's persona, tone, and base posture for every project and conversation -- it is the first block of the system prompt. Edit it to rebrand the assistant; delete it and the runtime regenerates a sensible default on next start. No env var, fixed path, and read-only hosts (containers) fall back to the built-in default instead of failing to boot - MCP Integration: connect external tools via Model Context Protocol (Claude Desktop-compatible config)
- MCP Apps: interactive UI rendered in sandboxed iframes for MCP tools with
_meta.uiresources - RAG Pipeline: PDF ingestion with ChromaDB vector search, Azure OpenAI embedding, and source citations
- Batch Processing: async job queue via Core MCP Server with real-time MCP Apps dashboard
- Multi-provider, multi-model switching: switch between Azure OpenAI and Anthropic (Claude) models mid-conversation. Add Claude models with
ANTHROPIC_MODELSand they appear in the existing model selector. Both Anthropic hostings supported: Direct (Anthropic public API) and Foundry (Anthropic on Azure AI Foundry) - Per-message reasoning effort: a selector next to the model picker sets how hard the model reasons, per message. Allowed levels and the default follow the selected model and are served by the backend -- Azure OpenAI (
low/medium/high/xhigh, defaultmedium); Anthropic (low/medium/high/xhigh/max, defaultxhigh, via adaptive thinking). The chosen effort is shown next to the model name and per-turn token usage, and persists with the session - Session management: save, search, organize into folders, pin, archive, fork, rename
- Sidebar folders: assign a color from a preset palette (set on create or change later via the folder menu), reorder folders by drag-and-drop, and collapse folders by default (open/closed state remembered per device); the folder list self-heals if its saved color/order values are ever corrupted
- Background Responses: long-running agent timeout prevention with stream resumption
- Conversation compaction: long sessions are auto-compacted before each model call (sliding-window by default) so the agent keeps responding instead of failing at the context-window limit; disable with
COMPACTION_STRATEGY=none - Context window consumption display with warning levels
- Per-turn token usage display
- OpenAI-compatible API: expose agent as
/v1/responsesendpoint for external apps via OpenAI SDK - Unified API authentication: single
API_KEYBearer token protects/v1/responses, every write REST endpoint, and the AG-UI chat stream for non-loopback (LAN) callers; same-machine clients always bypass - Web SPA authentication (optional): single-user ID/PW login with HttpOnly opaque session cookie for cloud-deployed instances; coexists with
API_KEY - Azure OpenAI credential lane choice: four lanes via one helper --
AZURE_OPENAI_API_KEY(api-key bypass, works everywhere) or Entra ID viaAZURE_CREDENTIAL_MODE=cli(default;AzureCliCredential/az login) /managed-identity(ManagedIdentityCredentialfor Azure App Service, Container Apps, AKS, Functions, VM) /default(DefaultAzureCredentialauto-discovery chain). One INFO log line per process announces the active lane at first credential resolution; no key value is ever logged. Cloud deployment guide atassets/docs/guides/azure-cloud-deploy.md. - CLI Client: chat, session/template/model management, TTS, and upload from the command line with local preflight validation for filename, MIME type, and size
.envupgrade tooling:chatwalaau env diffreports settings added / removed since your.envwas generated, andchatwalaau env syncre-renders.envto the new release's template (keys, comments, order) while preserving your values and keeping a timestamped backup- HTTPS/TLS support for LAN access with Secure Context (mkcert recommended)
- Multilingual chat with browser auto-translation suppressed
- Three layout scenarios: Chat, Popup, Sidebar
Architecture
The platform connects the UI and agent runtime through the AG-UI protocol.
Development Setup
Prerequisites
| Tool | Version | Install |
|---|---|---|
| Node.js | 20.19+ / 22.12+ | https://nodejs.org/ |
| pnpm | 10+ | npm install -g pnpm |
| Python | 3.12+ | https://www.python.org/ |
| uv | 0.9+ | https://docs.astral.sh/uv/ |
| Azure CLI | 2.x | https://learn.microsoft.com/cli/azure/install-azure-cli |
1. Azure Authentication
The backend supports four Azure OpenAI credential lanes selected by two environment variables. Pick the one that matches your deployment surface.
| Lane | When to use | .env setting |
|---|---|---|
| api-key | First-run / PoC / CI / container; cross-tenant Azure OpenAI | AZURE_OPENAI_API_KEY=<key> |
| cli (default) | Localhost development with az login available | AZURE_CREDENTIAL_MODE=cli (or unset) |
| managed-identity | Azure App Service, Container Apps, AKS, Functions, VM | AZURE_CREDENTIAL_MODE=managed-identity |
| default | One image targeting multiple surfaces | AZURE_CREDENTIAL_MODE=default |
Precedence: AZURE_OPENAI_API_KEY beats AZURE_CREDENTIAL_MODE. One
INFO log line per process announces the active lane on first credential
resolution; the key value is never logged.
Option A: az login (default, localhost)
The backend authenticates via AzureCliCredential. Log in before
starting.
az login
az account set --subscription <subscription-id>
Option B: AZURE_OPENAI_API_KEY (no Azure CLI required)
For first-run / PoC / CI / container scenarios where az login is
inconvenient, set the API key from your Azure OpenAI resource in
.env:
AZURE_OPENAI_API_KEY=<your-api-key>
When set, every Azure OpenAI client in the backend (MAF chat, Whisper STT, image generation, RAG embed / search) authenticates with the key instead of an Entra ID credential.
Option C: Managed Identity (Azure cloud deployments)
For ChatWalaʻau deployed to Azure App Service / Container Apps / AKS /
Functions / VM, assign a Managed Identity to the compute and grant it
the Cognitive Services OpenAI User role on the Azure OpenAI
resource scope, then set:
AZURE_CREDENTIAL_MODE=managed-identity
# AZURE_CLIENT_ID=<user-assigned-MI-client-id> # only if using user-assigned MI
User-assigned identities require AZURE_CLIENT_ID per the
azure-identity SDK convention. AKS workloads using federated
identity should set AZURE_CREDENTIAL_MODE=default instead so the
SDK's WorkloadIdentityCredential is picked up automatically.
Option D: default (DefaultAzureCredential auto-discovery)
For container images that target multiple deployment surfaces
(developer box, staging container, AKS pod) without changing the
.env, set:
AZURE_CREDENTIAL_MODE=default
DefaultAzureCredential is constructed with the interactive browser
excluded so headless cloud containers never pop a sign-in window. The
chain order is: Environment -> WorkloadIdentity ->
ManagedIdentity -> SharedTokenCache -> AzureCli ->
AzurePowerShell -> AzureDeveloperCli. The chatwalaau CLI
suppresses its az account show precheck automatically whenever the
active lane is not cli.
See
assets/docs/guides/azure-cloud-deploy.mdfor the full cloud deployment walk-through (RBAC role assignment, AKS Workload Identity setup, troubleshooting).
2. Backend Setup
Windows (PowerShell):
cd backend
copy .env.sample .env
# Edit .env and set your Azure OpenAI endpoint
notepad .env
uv sync --prerelease=allow
macOS / Linux:
cd backend
cp .env.sample .env
# Edit .env and set your Azure OpenAI endpoint
nano .env
uv sync --prerelease=allow
.env configuration (required):
AZURE_OPENAI_ENDPOINT=https://<your-resource>.openai.azure.com/
AZURE_OPENAI_MODELS=gpt-4o
3. Frontend Setup
cd frontend
pnpm install
4. Start Development Servers
Open two terminals:
Terminal 1 -- Backend:
cd backend
uv run uvicorn app.main:app --reload --app-dir src
Backend starts at http://localhost:8000
Terminal 2 -- Frontend:
cd frontend
pnpm dev
Frontend dev server starts at http://localhost:5173 (API requests are proxied to the backend)
5. Production Build
cd frontend
pnpm build
cd ../backend
uv run uvicorn app.main:app --app-dir src
The backend serves both frontend build artifacts and the API at http://localhost:8000
CLI Usage
Server Commands
chatwalaau Start the server
chatwalaau init Generate .env from template
chatwalaau init --force Overwrite existing .env
chatwalaau env diff Report .env drift vs the bundled template
chatwalaau env sync Re-render .env to the template (dry-run)
chatwalaau env sync --write Apply (creates a timestamped backup first)
chatwalaau hash-password Generate AUTH_PASSWORD_HASH (interactive)
chatwalaau hash-password --stdin --quiet Generate hash from stdin (scripted)
chatwalaau --host 0.0.0.0 Bind to all interfaces
chatwalaau --port 9000 Use custom port
chatwalaau --skip-auth-check Skip Azure CLI login check
chatwalaau --ssl-certfile cert.pem \
--ssl-keyfile key.pem Enable HTTPS (LAN access)
chatwalaau --version Show version
Client Commands
Interact with a running ChatWalaʻau instance from the command line. All client commands support --json for machine-readable output and --base-url / --api-key for remote server access.
# Chat with the agent (single-shot)
chatwalaau chat "What is the weather in Tokyo?"
# Interactive chat (REPL mode)
chatwalaau chat -i
# Chat with specific model and session
chatwalaau chat "hello" -m gpt-4o -s <session-id>
# Session management
chatwalaau sessions list
chatwalaau sessions get <id> --messages
chatwalaau sessions delete <id>
chatwalaau sessions export <id> -o backup.json
# Template management
chatwalaau templates list
chatwalaau templates create -n "Bug Report" -c "Describe the bug..."
# Model info
chatwalaau models list
# Text-to-Speech
chatwalaau tts "Hello world" -o greeting.mp3
# File upload
chatwalaau upload document.pdf -s <session-id>
# JSON output for scripting / agent-to-agent
chatwalaau sessions list --json | jq '.[].thread_id'
# Remote server with HTTPS (self-signed cert)
chatwalaau sessions list --base-url https://192.168.1.10:8000 --no-verify
chatwalaau upload validates the local file before sending the request: the file must exist, the sanitized filename must remain valid, the MIME type must be one of the supported image formats or PDF, and the size limit must stay within 20MB for images or 50MB for PDFs.
Environment variables for client configuration:
CHATWALAAU_URL=http://localhost:8000 # Default server URL
CHATWALAAU_API_KEY=sk-your-key # Bearer token (reuses API_KEY if not set)
Tech Stack
| Layer | Technology | Purpose |
|---|---|---|
| Frontend | React 19 + TypeScript + Vite | UI framework |
| Frontend | Tailwind CSS + shadcn/ui | Styling + Components |
| Frontend | Biome | Format + Lint |
| Backend | FastAPI + Python 3.12+ | API server |
| Backend | Microsoft Agent Framework | Agent execution + Tool control |
| Backend | Ruff | Format + Lint |
| Package | uv | Python dependency management |
| Package | pnpm | Node.js dependency management |
Optional Features
Anthropic Provider (Claude)
Enable Anthropic Claude models alongside Azure OpenAI -- they appear in the same model selector, can be picked per turn, and per-assistant-message regenerate works across providers. Default state is "Anthropic disabled" so upgrading from 0.65.0 with ANTHROPIC_MODELS unset is a no-op.
Two hostings, selected by one variable:
# direct (Anthropic public API) | foundry (Anthropic on Azure AI Foundry)
ANTHROPIC_HOSTING=direct
ANTHROPIC_MODELS=claude-sonnet-4-5-20250929,claude-haiku-4-5
Direct hosting (typical):
ANTHROPIC_API_KEY=sk-ant-...
# Optional override; useful for a corporate gateway or proxy
# ANTHROPIC_BASE_URL=https://your-gateway.example.com
Foundry hosting (Anthropic on Azure AI Foundry).
Endpoint -- supply exactly ONE (they are mutually exclusive). ANTHROPIC_FOUNDRY_RESOURCE is the Azure AI Services resource name (the subdomain) -- not the Foundry project URL or project name -- and expands to https://<resource>.services.ai.azure.com/anthropic/. Use ANTHROPIC_FOUNDRY_BASE_URL to give that full URL directly instead.
Auth -- pick ONE:
ANTHROPIC_HOSTING=foundry
ANTHROPIC_FOUNDRY_RESOURCE=my-aifoundry # resource name only (subdomain)
# OR: ANTHROPIC_FOUNDRY_BASE_URL=https://my-aifoundry.services.ai.azure.com/anthropic/
# Auth option A -- API key (sent as the api-key header):
ANTHROPIC_FOUNDRY_API_KEY=<foundry-key>
# Auth option B -- Entra ID (Azure CLI / Managed Identity): leave the API key
# EMPTY. The runtime uses the same Azure credential lane as Azure OpenAI
# (AZURE_CREDENTIAL_MODE: cli | managed-identity | default; AZURE_TENANT_ID),
# sending an "Authorization: Bearer <token>" header. Do NOT place a token in
# ANTHROPIC_FOUNDRY_API_KEY -- it is sent verbatim as the api-key header (HTTP 401).
Generation and extended thinking (Anthropic requires max_tokens on every request as a hard output cap):
ANTHROPIC_MAX_TOKENS=8192
Claude Opus 4.7/4.8 use adaptive thinking plus an effort level (output_config.effort); the deprecated budget_tokens mechanism is no longer used. The reasoning effort is chosen per message in the UI (default xhigh) -- there is no environment variable for it. See Reasoning effort.
Hosted web search is included for Anthropic models out of the box (uses Anthropic's web_search_20250305 tool, no extra config required). Every other agent feature -- Weather, Coding tools + Tool Approval, RAG, Image Generation, Vision input, MCP -- works on either provider as long as the chosen model supports tool calling. Speech-to-text, text-to-speech, image generation, and RAG embedding run on their own dedicated Azure models and are independent of the chat provider choice.
Prompt Templates
Save and reuse prompt templates from the chat interface:
TEMPLATES_DIR=.templates
- Click + button > Use template to open the management modal
- Create, edit, delete templates with name, category, and body
- Insert to Chat pastes the template into the input (editable before send)
- Click the FileText icon on any user message to save it as a template
Templates are stored as individual JSON files in the configured directory.
Image Generation
Generate and edit images via Azure OpenAI gpt-image-1.5:
IMAGE_DEPLOYMENT_NAME=gpt-image-1.5
- generate_image: create images from text prompts with configurable size, quality, format, background, and count (1-4)
- edit_image: modify existing session images using text prompts (prompt-based)
- Canvas Mask Editor: click the Edit button on any generated image to open a full-screen mask editor
- Draw over areas to edit with brush tools (S/M/L), eraser, undo/redo
- Enter a prompt and click Generate -- the agent edits only the masked region
- Generated images displayed inline in chat with click-to-open full-size
- Images stored in session upload directory and persist across reloads
The agent automatically uses these tools when users request image creation or editing. No opt-in flag needed -- the feature activates when IMAGE_DEPLOYMENT_NAME is set.
Coding Tools
Enable AI-powered file operations and shell execution:
CODING_ENABLED=true
CODING_WORKSPACE_DIR=C:\path\to\workspace
# Optional: bound file_read output to protect memory + context window
# CODING_FILE_READ_MAX_BYTES=1048576 # 1 MiB default
file_read now stats the target before reading and caps the output at
CODING_FILE_READ_MAX_BYTES (default 1 MiB). When the cap or the line
limit is hit, the response ends with an explicit
[TRUNCATED BY BYTES: ...] / [TRUNCATED BY LIMIT: ...] marker that
tells the agent how to paginate with offset=N.
Tool Approval & Conversation Compaction (Agent Harness)
Two Microsoft Agent Framework harness primitives, configured entirely
via .env.
Tool approval gates destructive coding tools behind an operator
decision. By default (TOOL_APPROVAL_MODE=auto), the agent's first
bash_execute or file_write call shows an inline approval card with
Approve / Reject / Approve for this session; read-only tools
(file_read, file_glob, file_grep) never require approval.
# auto (default) -> gate bash_execute + file_write
# always -> gate every non-read-only tool
# skip -> disable approval (autonomous, like before)
TOOL_APPROVAL_MODE=auto
TOOL_APPROVAL_REQUIRE_LIST=bash_execute,file_write
TOOL_APPROVAL_TIMEOUT_SEC=300
Set TOOL_APPROVAL_MODE=skip for the pre-approval "no friction"
behaviour on a trusted single-user machine (analogous to Claude Code's
--dangerously-skip-permissions). While skip mode is active the SPA
shows a persistent "Tool approval is DISABLED" banner. Headless lanes
(OpenAI Responses API, the chatwalaau chat CLI, and DevUI) auto-approve
every request and log a WARNING per auto-approval.
Conversation compaction keeps long sessions within the model's context budget by compacting in-memory history before each model call (the on-disk session JSON is never altered):
# none | sliding-window (default) | selective-tool-call | tool-result
COMPACTION_STRATEGY=sliding-window
COMPACTION_KEEP_LAST_GROUPS=4
COMPACTION_PRESERVE_SYSTEM=true
Text-to-Speech
Enable on-demand TTS for messages. Pick the provider with TTS_PROVIDER
(default elevenlabs).
Option A -- ElevenLabs:
TTS_PROVIDER=elevenlabs
ELEVENLABS_API_KEY=your-api-key
TTS_MODEL_ID=eleven_multilingual_v2
TTS_VOICE_ID=your-voice-id
Option B -- Azure OpenAI Realtime voice (e.g. gpt-realtime-2), reusing your
existing AZURE_OPENAI_ENDPOINT and credentials:
TTS_PROVIDER=azure-realtime
TTS_REALTIME_DEPLOYMENT=gpt-realtime-2
TTS_REALTIME_VOICE=alloy
#TTS_REALTIME_AUDIO_RATE=24000
Either way the speaker button plays audio and the download button saves an MP3 file; audio is cached to avoid duplicate API calls. The Azure Realtime lane reads the message text verbatim (it does not converse), matching ElevenLabs.
Agent Skills
Extend the agent with domain knowledge packages (Agent Skills specification):
SKILLS_DIR=.skills
Place SKILL.md files in subdirectories. The agent discovers and loads skills on demand:
.skills/
my-skill/
+-- SKILL.md # Required: instructions + metadata
+-- scripts/ # Optional: executable code
+-- references/ # Optional: documentation
+-- assets/ # Optional: templates, resources
Skills use progressive disclosure to minimize context window consumption (~100 tokens per skill when idle).
MCP Integration
Connect external tools and services via Model Context Protocol using the Claude Desktop-compatible configuration format:
# Optional: defaults to mcp_servers.jsonc.
# Set this to an explicit empty value to disable MCP.
# MCP_CONFIG_FILE=mcp_servers.jsonc
ChatWalaʻau ships with backend/mcp_servers.default.jsonc (git-tracked bundled default). The runtime resolves the operator override backend/mcp_servers.jsonc (gitignored) first, then falls back to the bundled default; no manual copy is needed on first run. To customise, create mcp_servers.jsonc and edit freely:
{
"mcpServers": {
"filesystem": {
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-filesystem", "/path/to/workspace"]
},
"remote-api": {
"url": "https://api.example.com/mcp",
"headers": { "Authorization": "Bearer token" }
}
}
}
- JSONC : the parser strips
//and/* */comments so operators can annotate entries inline. Strict-JSON contents are accepted unchanged. - stdio servers (with
command): ChatWalaʻau spawns the process and communicates via stdin/stdout - HTTP/SSE servers (with
url): ChatWalaʻau connects to a running remote server - MCP tools appear alongside built-in tools (Weather, Coding, Image Generation)
- Tool calls display with categorized icons: built-in tools have dedicated icons, Skills tools show BookOpen/FileText, MCP tools show Plug
- Server lifecycle managed automatically (startup/shutdown with zombie process prevention)
- Optional per-server fields (Claude Desktop-compatible, ignored elsewhere):
"load_prompts": true-- set when your MCP server implementsprompts/list(most community servers are tools-only; the default isfalseso filesystem, git, github, etc. connect cleanly out of the box)"load_tools": false-- skip thetools/listprobe for a tools-only server's prompts-only mode"request_timeout": 30-- per-call timeout in seconds forwarded to MAF
- Reuse your existing Claude Desktop / Claude Code / Cursor MCP configurations (strict-JSON files parse unchanged)
MCP Apps
MCP tools that declare a _meta.ui resource automatically render interactive UI within chat messages. The HTML View runs in a secure double-iframe sandbox with CSP enforcement.
# Optional: change the sandbox proxy port (default 8081)
# MCP_APPS_SANDBOX_PORT=8081
- Automatic discovery: UI-enabled MCP tools detected at server startup
- Double-iframe sandbox: Views run on a separate origin with no access to host DOM, cookies, or storage
- CSP enforcement: external resources blocked by default; servers declare required domains via metadata
- View-to-Server proxying: all View interactions proxied through the Host (auditable)
- Display modes: inline (in chat) and fullscreen
- Session persistence: View HTML stored as files for reload restoration
- Progressive enhancement: tools work as text-only when UI is unavailable or unsupported
No configuration needed -- MCP Apps activates when MCP tools have _meta.ui.resourceUri in their definitions. The sandbox proxy starts automatically alongside MCP servers.
RAG Pipeline
Upload PDF documents and ask questions about their content using vector similarity search:
CHROMA_DIR=.chroma
RAG_COLLECTION_NAME=default
RAG_TOP_K=5
EMBEDDING_DEPLOYMENT_NAME=text-embedding-3-small
RAG_CHUNK_SIZE=800
RAG_CHUNK_OVERLAP=200
# Optional: trailing-chunk merge threshold (unset -> RAG_CHUNK_SIZE // 4; 0 disables)
# RAG_CHUNK_MIN_SIZE=200
- Click + button > Attach PDF to upload a document
- Ask the agent: "Please ingest this document"
- The batch job processes: PDF parsing > chunking > embedding > ChromaDB storage
- Ask questions: "What does the document say about X?"
- The agent searches the knowledge base and responds with source citations (filename, page)
- ChromaDB PersistentClient: file-based vector storage (
.chroma/directory) - Azure OpenAI Embedding:
text-embedding-3-smallfor consistent multilingual quality - Singleton embedding client: Azure CLI credential resolution and TLS handshake run once per batch server process, not once per 100-text batch
- Overlap chunking: configurable chunk size (800 chars) and overlap (200 chars) with tail-merge that drops or folds sub-minimum trailing fragments so tiny chunks do not pollute top-k retrieval
- Metadata filtering: source filename, page number, chunk index for precise citation
- Deduplication: re-ingesting the same file overwrites existing chunks automatically
- PDF file cards: PDFs display as file icon cards (not image thumbnails) in chat
Requires the Batch Processing MCP Server to be configured (see below).
Batch Processing
Run long-running tasks (RAG ingestion, data pipelines) as background batch jobs with a real-time monitoring dashboard:
The bundled backend/mcp_servers.default.jsonc already registers the batch server. To customise (job dir, per-batch overrides), copy it to backend/mcp_servers.jsonc and edit:
- Start the server -- the batch MCP server launches automatically.
- Upload a PDF and ask the agent: "Please ingest this document".
- A real-time dashboard appears inline showing progress, status, and controls.
{
"mcpServers": {
"batch": {
"command": "uv",
"args": ["run", "python", "-m", "app.mcp_batch.server"],
// env intentionally empty: BATCH_JOBS_DIR and RAG_CHUNK_*
// live in backend/.env and are loaded by the batch subprocess
// via load_dotenv(override=False). Add keys here only to pin
// a per-batch override (highest precedence).
"env": {},
"load_prompts": false
}
}
}
- Conversation-based management: submit, monitor, cancel, delete jobs via chat
- MCP Apps dashboard: auto-refreshing progress bars, cancel/delete with confirmation dialogs
- File-based persistence: each job stored as a JSON file (crash-resilient)
- Registered job types:
rag-ingest(Operators add new job types underbackend/src/app/mcp_batch/jobs/) - Cooperative cancellation: jobs check cancel flag at each progress checkpoint
Background Responses
For long-running agent operations (e.g., o3/o4-mini reasoning models), enable Background Responses to prevent timeouts:
- Click the BG toggle button (left of the context window indicator)
- ChatInput border turns blue when active
- Continuation tokens are auto-saved to session for page reload resumption
No environment variable needed -- toggle on/off per session via the UI.
API Authentication
API_KEY is the unified Bearer token that protects the external-app
OpenAI API, every write REST endpoint, and the AG-UI chat stream when
reached from a non-loopback (LAN) client. Same-machine clients
(127.0.0.1, ::1, localhost) bypass auth so localhost development
stays zero-configuration even when APP_HOST=0.0.0.0 for LAN exposure.
API_KEY=sk-chatwalaau-your-secret-key-here
# APP_REQUIRE_AUTH_ON_LAN=true # default: fail-closed on LAN without a key
Decision matrix for write endpoints (image edit, upload, MCP Apps RPC,
sessions write, templates write, TTS, STT) and the AG-UI chat stream
(POST /ag-ui/):
| Client address | APP_REQUIRE_AUTH_ON_LAN | API_KEY | Outcome |
|---|---|---|---|
| loopback | any | any | allow |
| LAN | false | any | allow (operator opt-out) |
| LAN | true | empty | 503 |
| LAN | true | set | Bearer required |
/v1/responses (OpenAI API below) always requires a matching Bearer key regardless of client address because it is designed for external apps.
Upgrading from a release before v0.47.0 with APP_HOST non-loopback
and API_KEY unset: AG-UI now returns the same 503 / 401 as every other
write endpoint. Add API_KEY=... to .env, or accept LAN exposure
explicitly with APP_REQUIRE_AUTH_ON_LAN=false.
Web SPA Authentication
For deploying ChatWalaʻau as a private cloud web app where a single
operator signs in through the browser. Coexists with API_KEY (which
remains for CLI / external SDK access) and is disabled by default --
operators who do not set AUTH_USERNAME see no behavior change.
AUTH_USERNAME=admin
AUTH_PASSWORD_HASH=scrypt$N=16384,r=8,p=1$<base64-salt>$<base64-hash>
# AUTH_SESSION_TTL_SECONDS=86400 # default 24h, sliding
# AUTH_COOKIE_SECURE=auto # auto / true / false
# AUTH_COOKIE_NAME=chatwalaau_session
Generate AUTH_PASSWORD_HASH with the bundled CLI:
# Interactive (prompts twice for confirmation, hidden input)
chatwalaau hash-password
# From a secret manager / pipeline
echo "$PASSWORD" | chatwalaau hash-password --stdin --quiet
When AUTH_USERNAME is set, the SPA renders a /login page; the
server validates credentials in constant time, issues an opaque
token, and returns it via an HttpOnly + SameSite=Strict cookie.
The backend then accepts EITHER a Bearer API_KEY OR a valid
session cookie on every write endpoint and the AG-UI chat stream.
The /v1/responses external-app path stays Bearer-only.
Properties:
- No new Python dependency (uses stdlib
hashlib.scryptandsecrets) - Single-user model: one username + one password hash in
.env - Process-local session store: no on-disk persistence, restarts re-prompt the user
- HTTPS strongly recommended for non-loopback deployments
- Loopback CLI calls (
curl localhost,chatwalaausubcommands) keep their no-credential bypass so the local development workflow is unaffected
OpenAI Compatible API
Expose the agent as an OpenAI-compatible endpoint for external applications:
API_KEY=sk-chatwalaau-your-secret-key-here
Any app using the OpenAI SDK can consume the agent by pointing base_url:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="sk-chatwalaau-your-secret-key-here",
)
# Non-streaming
response = client.responses.create(
model="chatwalaau",
input="What is the weather in Tokyo?",
)
# Streaming
stream = client.responses.create(
model="chatwalaau",
input="Explain quantum computing.",
stream=True,
)
for event in stream:
if event.type == "response.output_text.delta":
print(event.delta, end="", flush=True)
- All agent Tools (Weather, Coding, Image Generation) and Skills are available
- Multi-turn conversations via
previous_response_id - API sessions appear in the chat sidebar with an API badge
- Streaming (SSE) and non-streaming response modes
- For HTTPS/LAN access, see OpenAI API Setup Guide
Voice Input via Realtime STT
ChatWalaʻau supports two transcription transport paths for the
POST /api/transcribe endpoint, selected automatically by the
configured deployment name:
# REST audio.transcriptions path (default, classic Whisper / gpt-4o-transcribe / gpt-4o-mini-transcribe)
WHISPER_DEPLOYMENT_NAME=whisper
# Realtime API WebSocket path (gpt-realtime-whisper)
WHISPER_DEPLOYMENT_NAME=gpt-realtime-whisper
WHISPER_REALTIME_CONNECTION_DEPLOYMENT=gpt-realtime-mini
REST path (zero-config)
Set WHISPER_DEPLOYMENT_NAME to a deployment of whisper-1,
whisper, gpt-4o-transcribe, or gpt-4o-mini-transcribe. The
backend calls POST /audio/transcriptions synchronously and
returns the transcript. No extra deployments needed.
Realtime path (gpt-realtime-whisper)
Per Microsoft Learn realtime-audio-websockets, the Realtime API
URL ?model= query accepts only VOICE Realtime deployments
(gpt-realtime / gpt-realtime-mini / gpt-realtime-1.5).
gpt-realtime-whisper is a transcription-only model and runs
alongside a voice model. Two deployments are required:
- Deploy a voice Realtime model in Azure Foundry, e.g.
gpt-realtime-mini(cheapest). - Deploy
gpt-realtime-whisperfor transcription. - Set both in
.env:WHISPER_DEPLOYMENT_NAME=gpt-realtime-whisper WHISPER_REALTIME_CONNECTION_DEPLOYMENT=gpt-realtime-mini
Transport is auto-selected by the WHISPER_DEPLOYMENT_NAME
substring (realtime -> Realtime, otherwise REST). Override with
WHISPER_MODEL_KIND=rest|realtime if your deployment name
violates the convention.
Optional knobs (defaults work out of the box):
# Empty default selects the GA URL path /openai/v1/realtime?model=...
# Set this to a preview value (e.g. 2025-04-01-preview) only for
# legacy models such as gpt-4o-realtime-preview.
AZURE_OPENAI_REALTIME_API_VERSION=
# Browser webm/Opus is resampled to this PCM rate server-side.
# Allowed: 16000, 24000 (24000 is the OpenAI Realtime default).
WHISPER_REALTIME_AUDIO_RATE=24000
No SPA / useVoiceInput change -- the POST /api/transcribe
contract is byte-for-byte identical across both transports.
HTTPS / LAN Access
Access ChatWalaʻau from other devices on your home network (phones, tablets, other PCs). HTTPS enables browser Secure Context for voice input and clipboard on non-localhost origins.
APP_HOST=0.0.0.0
APP_SSL_CERTFILE=.certs/cert.pem
APP_SSL_KEYFILE=.certs/key.pem
Setup:
- Install mkcert and run
mkcert -install - Issue a certificate:
mkcert -cert-file .certs/cert.pem -key-file .certs/key.pem <your-ip> localhost 127.0.0.1 - Set the env vars above in
.env - Allow ports through firewall (8000 for production, 5173 for dev mode)
- Install the CA certificate (
rootCA.pem) on each client device
Access from LAN: https://<your-ip>:8000
When SSL is not configured, the server runs in HTTP mode as usual (no breaking change).
Multi-Model Switching
Switch between OpenAI-family models mid-conversation:
AZURE_OPENAI_MODELS=gpt-4o,o3,gpt-4.1-mini
- Model selector dropdown appears above the chat input (hidden when only one model configured)
- Per-session model selection persisted across page reloads
- Regenerate with different model: click the chevron on the Regenerate button to choose a model
- Per-message model label: each assistant message shows which model generated it
- All models share the same Tools, Skills, and MCP integrations
Reasoning effort is selectable per message from a selector next to the model picker. The allowed levels and the default follow the selected model and are served by the backend (GET /api/model); the chosen effort is shown next to the model name + token usage and persists with the session. There is no environment variable -- the per-provider default is fixed (Azure OpenAI medium; Anthropic xhigh):
- Azure OpenAI (gpt-5.5 / gpt-5.4):
low,medium,high,xhigh->reasoning.effort - Anthropic (Opus 4.8 / 4.7):
low,medium,high,xhigh,max-> adaptive thinking +output_config.effort
Per-model context window limits (one shared variable across all providers):
MODEL_MAX_CONTEXT_TOKENS=gpt-5.5:1050000,gpt-5.4:1050000,claude-opus-4-8:200000,claude-opus-4-7:200000
Context Window
The progress bar above the chat input shows context window consumption rate. Colors change at 80% (amber) and 95% (red). When multiple models are configured, the display updates automatically when switching models.
DevUI
Enable Microsoft Agent Framework DevUI for debugging:
DEVUI_ENABLED=true
DEVUI_PORT=8080
# DevUI runs in a daemon thread with its own asyncio event loop.
# Loop-bound handles (MCP tool async contexts, ChromaDB / SQLite) are
# excluded from the DevUI agent by default. Opt in with false.
# DEVUI_DISABLE_MCP=true
# DEVUI_DISABLE_RAG=true
Access at http://localhost:8080
DevUI receives a dedicated Agent instance that reuses the main agent's
function tools, skills, and model client but omits MCP tools and
rag_search by default. This avoids cross-loop invocation between the
DevUI daemon thread and the main FastAPI event loop.
Corporate Networks (TLS-Intercepting Proxy)
Operators behind a corporate TLS-intercepting proxy (Zscaler, Netskope, on-prem SSL middlebox) typically hit the following on the first outbound HTTPS call from the backend (Azure AD token acquisition, Azure OpenAI Responses, ElevenLabs, Open-Meteo, etc.):
httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: self-signed certificate in certificate chain
The root cause is that Python's bundled certifi store does not know
the in-house root CA that the proxy re-signs traffic with. The
recommended remedy is to install the opt-in corp extras:
pip install "chatwalaau[corp]"
This pulls pip-system-certs, which routes Python's TLS validation
through the host OS certificate store (where the corporate root CA is
already trusted). No application source change, no env var change,
no behaviour change for non-corporate environments.
Alternative remedies (use whichever is available in your environment):
# Point Python at an explicit CA bundle
export SSL_CERT_FILE=/path/to/corp-root-ca.pem
export REQUESTS_CA_BUNDLE=/path/to/corp-root-ca.pem
Upgrading .env Across Releases
Most releases add value through new opt-in settings that default off,
so the server keeps working after pip install -U chatwalaau with no
.env edits. But you cannot tell from your own .env which new settings
became available, and old keys pile up. Two offline commands reconcile
your .env against the template bundled with the installed release:
# See what this release added (new settings) or removed (no longer read)
chatwalaau env diff
chatwalaau env diff --json # machine-readable, for scripts
# Preview the reconciliation as a unified diff (writes nothing)
chatwalaau env sync
# Apply: re-render .env to the template's keys / comments / order,
# preserving your values, after writing a timestamped backup
chatwalaau env sync --write
- Your values are preserved verbatim (quoting,
export, inline comments included); only the layout and per-key documentation are refreshed to match the installed release. - Nothing is deleted. Keys in your
.envthat the template no longer has (removed or custom) are moved into anUnmanaged keyssection. - A timestamped backup (
.env.<UTC>.bak) is always written before--write, and never overwrites a prior backup. env syncis a dry-run by default; review the diff before applying.- On startup, the server logs one line when your
.envis missing keys the installed release added, pointing you tochatwalaau env diff.
Note:
env sync --writereplaces hand-written comments on template-managed keys with the template's comments (your values stay). Custom keys are preserved in theUnmanaged keyssection, and the backup lets you recover anything.
Supported Platforms
- Windows 10/11
- macOS (Intel / Apple Silicon)
- Linux (Ubuntu, Debian, etc.)