convez.ai
MCP ServerAutomaçãoAvançado
PascalAI2024

JarvisNano — Agente IA Desktop em Microcontrolador

PascalAI2024/JarvisNano

Agente IA em microcontrolador: voz, câmera, Lua skills e LLM local

Ver no GitHub
BomPluginScore · de 0 a 100

MCP server que transforma placas ESP32-S3 em agentes de IA autônomos com captura de voz, visão computacional e execução de skills em Lua. Roda modelos como Gemma e MiniMax-M2.7 em hardware de $14, com interface web integrada e suporte a OpenAI, Anthropic e outras APIs de LLM.

Instalação

$npx convez-ai PascalAI2024/JarvisNano

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
9
Forks
0
Acompanhando
1
Saúde
100%
Último commit
há 1 semana
Versão
—
Issues abertas
1
Contribuidores
2

Casos de uso

  • Assistente de voz embarcado para smart devices e IoT
  • Prototipagem de agentes de IA com restrições de recursos
  • Dispositivo de borda com inferência de linguagem e visão

Veredito

Projeto robusto que viabiliza agentes de IA em hardware ultra-restrito (ESP32-S3). Excelente para quem busca IA de borda com controle total, mas requer experiência em firmware embarcado e sistemas de baixo nível.

README

A tradução para o português deste README ainda está na fila.


JarvisNano is a board adaptation + reference build that brings Espressif's ESP-Claw "Chat Coding" agent framework to small, accessible ESP32-S3 hardware. Two boards are currently supported in-tree:

BoardVendorStrengthsAdaptation at
XIAO ESP32-S3 SenseSeeed StudioSmallest S3 with on-board PDM mic + camera (OV3660 / OV2640)boards/seeed/xiao_esp32s3_sense/
ESP32-S3-Touch-AMOLED-1.75Waveshare1.75" round AMOLED, capacitive touch, dual MEMS mic + ES7210 AEC, 16 MB flashboards/waveshare/esp32s3_touch_amoled_1_75/

They share one firmware tree, one dashboard, one Android companion. Pick the board whose strengths match your use case — the XIAO if you want voice + vision, the AMOLED if you want a screen + AEC mic array. End-to-end agent loop has been verified on both.

The bare board, plugged into USB-C, gives you:

  • 🎙️ on-device PDM mic capture (full-duplex on I²S0)
  • 💬 LLM-driven chat (OpenAI / Anthropic / MiniMax-M2.7 / Qwen / DeepSeek / custom)
  • 🧠 dynamic Lua skill loading (no reflash to teach new behaviors)
  • 🟧 visible heartbeat on the on-board user LED — boot flash + idle wink
  • 🔌 MCP server + client over LAN
  • 🛜 self-hosted Wi-Fi provisioning portal on first boot
  • 🎛️ browser admin / onboarding / WebSerial flasher (no esptool install)

Add a PAM8002A analog speaker, a 503450 LiPo, and a 1.28" Seeed Round Display, and the same firmware grows into a battery-powered desktop concierge.


Meet The J.A.R.V.I.S. — Voice Companion


System at a glance

flowchart LR
    U[You]

    subgraph XIAO[XIAO ESP32-S3 Sense]
        MIC[PDM mic RX<br/>GPIO41/42 on I2S0]
        I2S[PDM-TX speaker<br/>GPIO4 + RC LPF on I2S0]
        LED[User LED GPIO21]
        CAM[OV3660 / OV2640 on-board]
        I2C[I2C bus GPIO5/6]
        CORE[ESP-Claw runtime<br/>Lua + Agent loop]
        WIFI[Wi-Fi STA + AP]
    end

    subgraph Cloud[Phase 1 cloud LLM]
        LLM[LLM API<br/>MiniMax-M2.7 / GPT / Claude]
        IM[Telegram / Web IM]
    end

    subgraph Future[Phase 2 / 3 local + visual]
        AMP[PAM8002A combo amp<br/>+ speaker + LiPo]
        SCR[Seeed Round Display<br/>1.28in GC9A01]
        BLE[Phone over BLE<br/>Gemma 4 E4B local]
    end

    U -->|voice| MIC
    MIC --> CORE
    CORE --> LED
    CORE -->|prompt + tools| LLM
    LLM -->|reply| CORE
    CORE -->|TTS PDM| I2S
    I2S -.->|Phase 2| AMP
    AMP --> U
    CORE <-->|chat| IM
    IM <--> U
    CORE -->|mDNS http| WIFI
    I2C -.->|Phase 3| SCR
    CORE <-.->|Phase 3| BLE
    CAM -.-> CORE

Why this exists

The official esp-claw repository ships board configs for the M5Stack CoreS3, LilyGo T-Display-S3, DFRobot K10, and a few Espressif breadboards — but not the XIAO ESP32-S3 Sense, the cheapest accessible S3 board with a built-in microphone.

This repo:

  1. Provides the board adaptation at boards/seeed/xiao_esp32s3_sense/ — board info, peripherals (full-duplex PDM on I²S0), devices, sdkconfig, setup C.
  2. Patches an upstream codegen bug that broke ESP32-S3 builds.
  3. Ships a Docker-only build/flash flow so you don't need to install ESP-IDF locally.
  4. Adds a native status LED heartbeat on GPIO21 that gives the physical board a boot flourish and alive pulse without consuming Lua heap.
  5. Provides a browser admin (dashboard/) — three-tab cockpit + onboarding wizard + ESP Web Tools flasher.
  6. Provides an open-source Android companion (android/) — Kotlin + Compose, mDNS discovery, BLE skeleton, Gemma 4 stub.
  7. Ships four parametric 3D-printable enclosures (hardware/enclosure/) with mermaid plans + dimensioned drawings.
  8. Documents the public protocol (docs/PROTOCOL.md) so any client (web, Android, ZeroChat, third-party) can integrate cleanly.

Quick start

You need: Docker, the XIAO Sense, a USB-C cable, and Python 3.

git clone git@github.com:PascalAI2024/JarvisNano.git
cd JarvisNano

# 1. Bootstrap: clones esp-claw, applies patches, copies board + FATFS assets
./scripts/bootstrap.sh

# 2. Build the firmware in Docker (~10 min the first time)
./scripts/bootstrap.sh build

# 3. Plug in the XIAO and flash from the host
./scripts/flash.sh

Or no-CLI install: open dashboard/index.html in Chrome / Edge → Flash tab → click Install → ESP Web Tools programs the merged .bin over WebSerial. No esptool, no Docker for the user.

Then plug in, open the captive Wi-Fi esp-claw-XXXXXX, visit http://192.168.4.1/ (or dashboard/onboard.html) to set Wi-Fi + LLM credentials.

Detailed walk-through in docs/BUILD.md.

Current hardware-session handoff lives in docs/NEXT_SESSION.md. Read it first before debugging the connected Waveshare AMOLED board; the USB reset path has a specific trap.

The current Phase-2 finish order is tracked in docs/FINISH_PLAN.md, with the full wave task board in docs/PHASE2_TASKS.md.


Cockpit — browser admin + onboarding + flashing

dashboard/index.html is a single-file vanilla-JS admin that talks straight to the device. Three tabs:

TabWhat it does
CockpitLive system telemetry, Wi-Fi + LLM state, mic / camera / battery / Bluetooth tiles, full chat with the agent (with <think> reasoning collapsed), capability + Lua module index, FATFS browser, live event stream, restart + new-session controls.
FlashOne-click ESP Web Tools flasher — Chrome / Edge users program a fresh XIAO Sense over USB-C from the browser, no esptool install. Manifest at dashboard/firmware/manifest.json.
SettingsEditable form for every device config field (Wi-Fi, LLM provider, IM credentials). Saves the diff back via /api/config. Eye-toggle for sensitive fields.

Plus dashboard/onboard.html — a 5-step first-boot wizard (Welcome → Wi-Fi → LLM preset → test round-trip → Done) with 6 LLM presets aligned to the firmware schema.

Open dashboard/index.html from any local server (python3 -m http.server in dashboard/) and point it at esp-claw.local. Add ?demo=1 to mask SSIDs / IPs / API key for clean screenshots.


Companion apps

AppRepoPurpose
android/this repo, android/Open-source Kotlin + Jetpack Compose reference companion. Cockpit / Chat / Settings / About screens, mDNS discovery, BLE GATT skeleton (Phase 2), Gemma 4 E4B local-LLM interface (Phase 3). See android/README.md.
Protocol clientsany repoThe public HTTP/WebSocket/MCP/BLE contract is documented in docs/PROTOCOL.md so web, Android, React Native, desktop, and third-party clients can integrate without copying firmware internals.

Both clients compute the same five canonical Phase-2 BLE GATT UUIDs from a single uuidv5 namespace. The UUID contract is shipped; the firmware GATT service is still planned.


Current hardware status

The USB-connected XIAO enumerates on macOS as /dev/cu.usbmodem* and has been flashed with the 8 MB ESP-Claw image. On boot it brings up FATFS, Wi-Fi STA/AP fallback, router rules, scheduler, Lua runtime, Web IM, and the MCP server. The current XIAO build intentionally disables the App Claw interactive serial REPL because ESP-IDF console command registration can exhaust internal heap after all Phase-2 services are started. USB serial logging remains enabled for boot diagnostics.

The physical GPIO21 LED heartbeat is now native firmware in edge_agent/main.c, patched by scripts/bootstrap.sh during bootstrap. It starts after the core services are alive so the event router and scheduler get first claim on heap, and LED allocation is non-fatal. The older Lua status_led.lua prototype remains in FATFS for future state-pattern work, but the boot_status_led router rule is disabled on XIAO builds so Lua heap is not spent at boot.

Reference boot log

I (...) BOARD_MANAGER: All peripherals initialized
I (...) app: FATFS mounted total=1482752 used=389120
I (...) app: Wi-Fi STA ready: 192.168.x.x
I (...) claw_event_router: Loaded 7 router rules
I (...) cap_lua_rt: Lua runtime ready: scripts=/fatfs/scripts registered_modules=17
I (...) cap_mcp_srv: MCP server ready: http://esp-claw.local:18791/mcp_server
I (...) claw_event_router: event router task started
I (...) app_claw: Publishing startup trigger event: startup/boot_completed
I (...) app: Native status LED on gpio 21 active_low=1
I (...) main_task: Returned from app_main()

LLM round-trip was previously verified end-to-end on hardware (XIAO ESP32-S3R8 / 8 MB octal PSRAM / IDF v5.5.4) with MiniMax-M2.7 over a custom OpenAI-compatible endpoint. The current flashed build is stable through service startup and the physical on-board LED alive effect. Remaining Phase-2 validation focus is LAN HTTP reachability from the dashboard/client network path, Android BLE diagnostics, TTS, battery ADC, and camera capture.


Hardware — desktop enclosure

Four parametric OpenSCAD enclosure concepts ship in hardware/enclosure/, each with a PLAN.md (mermaid front / top / cross-section views), a technical-drawing.svg (dimensioned multi-view engineering drawing), and a printable enclosure.scad.

Recommended: the Monolith — 78 × 68 × 66 mm, matte charcoal PETG / ASA, 4× M2 brass-insert screws, lid off in under 60 s, ~4.5 h print on a 0.4 mm nozzle. See hardware/enclosure/COMPARISON.md for the full matrix and pick rationale, and hardware/enclosure/assembly-flow.md for the assembly sequence.

The case is sized to fit the XIAO ESP32-S3 Sense, an audio amp + speaker module (PAM8002A combo or MAX98357A I²S — both fit), a 503040 / 503450 LiPo with foam shim, and a 39 mm round cutout from day one for the future Seeed Round Display for XIAO.


Roadmap

timeline
    title JarvisNano phases
    Phase 1 shipped : Board adaptation
                    : Full-duplex PDM on I2S0 (mic + speaker)
                    : MiniMax-M2.7 cloud LLM verified
                    : Wi-Fi provisioning + Telegram + Web IM
                    : Status LED heartbeat (GPIO21)
                    : Browser dashboard (Cockpit / Flash / Settings)
                    : 5-step onboarding wizard
                    : ESP Web Tools WebSerial flasher
                    : Public PROTOCOL.md
                    : Android companion scaffold
                    : ZeroChat integration PR
                    : 4 parametric 3D-printable enclosures
                    : Upstream esp_board_manager codegen PR
    Phase 2 partial : PDM-TX path + audio level endpoint
                    : Canonical BLE UUIDs
                    : PAM8002A analog amp + 28 mm speaker
                    : 503450 LiPo + on-board USB-C charger
                    : Real TTS + wake-word
                    : BLE GATT service (UUIDs in PROTOCOL.md)
                    : Battery + camera HTTP endpoints
    Phase 3         : Seeed 1.28in round AMOLED touchscreen
                    : LVGL chat UI + animated emote face
                    : Privacy Mode (phone runs Gemma 4 E4B over BLE)
    Phase 4         : OV3660 / OV2640 vision tools (describe / OCR)
                    : MCP server exposes desktop sensors
                    : Community remix release

Full detail in docs/ROADMAP.md.


Upstream bug fix

While bringing this board up, we found a bug in esp_board_manager's PDM RX codegen: it emits HP-filter struct fields (hp_en, hp_cut_off_freq_hz, amplify_num) for any chip with SOC_I2S_HW_VERSION_2, but those fields are gated by SOC_I2S_SUPPORTS_PDM_RX_HP_FILTER — only set on the ESP32-P4. ESP32-S3, S2, C3 fail to compile.


Layout

JarvisNano/
├── boards/seeed/xiao_esp32s3_sense/   # 5-file board adaptation (yaml + setup C)
├── firmware/                          # Lua skills baked into the FATFS image
│   ├── lua/status_led.lua             #   prototype state patterns; not auto-run on XIAO
│   └── router_rules/                  #   event-router rules
├── hardware/enclosure/                # 4 parametric OpenSCAD designs + drawings
├── dashboard/                         # browser admin (Cockpit + Flash + Settings + Onboard)
│   ├── index.html
│   ├── onboard.html
│   └── firmware/                      #   merged .bin + ESP Web Tools manifest
├── android/                           # Kotlin + Compose companion app
├── docs/                              # ARCHITECTURE · BUILD · HARDWARE · ROADMAP · PROTOCOL · RELEASE_CHECKLIST
├── images/                            # mascot · wordmark · renders · dashboard shots · early concepts
├── patches/                           # upstream fixes + generated ESP-Claw patch artifacts
├── scripts/                           # bootstrap.sh · flash.sh · smoke-build.sh
├── CONTRIBUTING.md                    # contribution workflow and verification matrix
├── SECURITY.md                        # vulnerability reporting and secret-handling policy
├── CHANGELOG.md                       # release notes
└── README.md

License

Apache-2.0 — same as upstream esp-claw.

Credits

Ver o original no GitHub

Bom para

Relacionados