Agent Utils
harryaskham/agent-utils
Toolkit completo para desenvolver agentes e extensions em Pi/JavaScript
Ver no GitHubColeção de utilitários, ferramentas e MCP servers para construir agentes e extensions em JavaScript. Fornece templates de prompts, ferramentas nativas para busca web, preview de imagens, orquestração de VMs Firecracker, agentes de voz em tempo real e auto-atualização de pacotes Pi.
Instalação
npx convez-ai harryaskham/agent-utils1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Construir agentes com busca web ao vivo e contexto real-time
- Orquestrar máquinas virtuais Firecracker para workloads isolados
- Implementar interfaces de voz com OpenAI Realtime
- Gerenciar auto-atualização e reload de ferramentas em tempo de execução
Veredito

Repositório maduro com foco em operações avançadas de agentes, especialmente para ambientes Pi. Bem documentado em arquitetura interna, mas exige familiaridade profunda com JavaScript, MCP servers e ecossistema Pi para aproveitamento máximo.
README
A tradução para o português deste README ainda está na fila.
agent-utils
Tools, skills, agents, MCP servers, etc
Cacophony workflow notes
Agent/operator bead-filing conventions live in docs/bead-workflow.md, including the distinction between parent epics and blocking dependencies.
Extension tool schema guidance lives in docs/extension-tool-schemas.md. Prefer the local ToolSchema helper for simple extension tools unless the package has a guaranteed runtime TypeBox dependency.
Pi extension capability boundaries (which builtin slash commands an extension can intercept via the input event vs. ones the TUI submit handler consumes first, and why getArgumentCompletions has no ctx so runtime state must be captured at session_start) live in docs/extension-capability-boundaries.md.
Pi package
This repo is also a Pi package.
After tagging a release, it can be installed with:
pi install git:github.com/harryaskham/agent-utils@v1
It currently provides:
/web-searchprompt templatesearch_webnative Pi tool for live web lookups via GitHub Copilot Responses APIkitty_image_preview_*native Pi tools for persistent terminal image previews via the kitty graphics protocolfirecracker_vm_*native Pi tools for preparing, spawning, inspecting, and stopping Firecracker VM workloads for Tendril-visible servicesxvfb_*native Pi tools (xvfb_ensure/xvfb_stop/xvfb_status) for opt-in, guarded Xvfb virtual-display orchestration on headless Linux nodes so display-dependent tools can run- OpenAI Realtime provider and voice commands (
/rt,/rt summary=true,/rt-listen,/rt-doctor,/rt-status full,/rt-off) viaextensions/realtime-agent.js. See the Realtime Agent guide for recommended workflows, Pulse/phone routing, summary context mode, VAD tuning, replay, and troubleshooting. /update,/reload-tools,/restart, and thepi_self_update/pi_reload_tools/pi_restartnative Pi tools viaextensions/pi-self-update.js./updaterunspi update --extensions, reloads the Pi runtime, then refreshes active tools; use/update --no-reloadto skip the reload and/update --statusto inspect startup auto-update controls. Startup package auto-update is opt-in: setPI_AUTO_UPDATE_ON_STARTUP=1or add{ "piSelfUpdate": { "autoUpdateOnStartup": true, "autoReloadAfterUpdate": true } }tosettings.json;PI_AUTO_RELOAD_AFTER_UPDATE=0orautoReloadAfterUpdate: falseleaves changed packages installed but asks the user to run/reloador/reload-tools, andPI_OFFLINE=1suppresses startup updates./reload-toolsreloads extensions/resources and then activates all registered tools so newly landed dynamic tools become model-visible./restartis stronger: it re-execs the Pi process against the current persisted session with--session, preserves recognized runtime flags, and deliberately omits original prompt and@filestartup arguments so non-idempotent launch instructions are not replayed while newly installed runtime/tool injections are picked up. Use/restart --dry-runto inspect the reexec plan. The tools queue the matching slash commands as follow-ups for agent-driven updates, refreshes, and restarts.- GitHub Copilot OAuth stale-token recovery via
extensions/copilot-auth-refresh.js: when Pi would otherwise surfaceNo API key for provider: github-copilot, the extension reloads auth storage and retries Copilot auth resolution. If the provider error still reachesagent_end, it reloads auth storage and queues one follow-up retry of the previous user request so managed agents do not stop on the transient missing-token state./copilot-auth-refreshmanually reloads the same auth storage without a full runtime reload. /effort [status|off|minimal|low|medium|high|xhigh|adaptive],/fast [on|off], and theself_set_effortnative Pi tool viaextensions/effort.js, shortcuts for Pi core thinking controls and configured fast-model variants./effortandself_set_effortdelegate every supported value, includingadaptive, directly to Pi's thinking-level API and never rewrite provider request payloads; adaptive-thinking request semantics and model metadata are owned by Pi itself.self_set_effortis agent-callable specifically for operator-instructed changes and its tool wording tells agents not to use it for autonomous self-tuning./fastswitches only between the current model id and a configured sibling with the same id plus or minus a-fastsuffix; it does not lower reasoning effort, enable adaptive thinking, or otherwise mutate request payloads./m <provider/model>and theself_set_modelnative Pi tool viaextensions/m.js, an independent model switcher that always resolves against the full model registry and switches immediately, with Tab-completion over every availableprovider/modelstring. Pi's built-in/modelcommand and Ctrl+P cycling are governed by the scoped-models feature, so when models are scoped (for example viaenabledModels)/model <arg>only matches the scoped set and arbitrary models fall through to the selector UI./mandself_set_modeldeliberately ignore that scope so an operator-instructed agent can jump straight to any model without opening the selector or toggling itsallscope; they never touch/modelor/scoped-models, which keep governing Ctrl+P cycling. Matching mirrors Pi core: canonicalprovider/id, then aprovider/idsplit, then an unambiguous bare id./mwith no argument prints usage and the available-model count.self_set_modelis agent-callable specifically for operator-instructed changes and its tool wording tells agents not to use it for autonomous self-tuning./true-defaults [status|apply]viaextensions/true-defaults.js, a persistence guard for Pi model/provider/thinking defaults. Configure immutable-by-convention values insettings.jsonunderagentUtils.trueDefaults, for example{ "agentUtils": { "trueDefaults": { "provider": "litellm-anthropic", "model": "claude-sonnet-4-5", "thinkingLevel": "medium" } } }; the extension copies them back to Pi's built-indefaultProvider,defaultModel, anddefaultThinkingLevelon startup and clean shutdown, then delegates runtime model/thinking application to Pi core unchanged. Runtime switching through/model, Ctrl+P,/settings, and/effortremains allowed; those runtime changes just do not become the persisted defaults unless the true-default values are edited directly./eink [on|off|status]viaextensions/pi-graphics.js, a tablet/e-ink toggle that applies the packagedeinktheme, transparent greyscale surfaces, cell cursor, static Unicode editor chrome, and minimal/no animation./eink offrestores the previous Pi graphics/theme settings saved before enabling e-ink mode.app_automation_*native Pi tools plus/tendril-appfor blessed Slack, canvas, Outlook, and Teams app automation diagnostics/overviews/freshness/plansskill-server/skill-search, a Rust CLI + MCP stdio meta-tool for dynamic skill and host MCP server discovery
App automation Pi extension
The app automation extension is loaded from extensions/app-automation.js. It gives agents a Pi-native catalog of blessed high-level actions for API-less web apps before they fall back to raw Playwright or Tendril commands.
Available tools:
app_automation_list— list configured apps and high-level actions.app_automation_doctor— diagnose catalog/state-root/Playwright CLI setup, Tendril remote/WSL bridge configuration, latest ms-dev CDP refresh status plus non-secret bridge config hints, optional bridge target-discovery probe, and standard action executability.app_automation_overview— summarize configured work apps, active refreshers, snapshot freshness, standard refresh-action freshness, latest snapshot digests, and optional compact snapshot links.app_automation_work_briefing— build a compact stale-aware shared briefing index from Slack, Outlook mail/calendar, Teams, and Calendar snapshots for natural-language questions.app_automation_personal_status— check personal automation prerequisites for a separate personal loop: verifies the existinggwsGoogle Workspace CLI auth state locally and optionally onms-dev, and scans~/org/todo.orgfor timely open items without adding a flake input or storing Google tokens.app_automation_msdev_cdp_refresh— refresh work-app snapshots throughms-devWindows Chrome CDP using the PowerShell WSL escape route when local Playwright/Tendril cannot reach Windows browser state. ConfigureAPP_AUTOMATION_MSDEV_SSH_TARGET=<user>@ms-dev; SSH/SCP connection attempts use bounded non-interactive connect options by default and can be tuned per call withsshConnectTimeoutSecondsor viaAPP_AUTOMATION_MSDEV_SSH_CONNECT_TIMEOUT_SECONDS. On flapping Tailscale/SSH links,preflightAttemptsorAPP_AUTOMATION_MSDEV_PREFLIGHT_ATTEMPTScan retry the initial SSH preflight before aborting the whole work-app pull; attempts are capped at five, and0skips preflight to try scp/PowerShell directly when the preflight itself is hanging. If scp is the unreliable stage,scriptTransfer: "inline"orAPP_AUTOMATION_MSDEV_SCRIPT_TRANSFER=inlineruns the generated PowerShell throughpwsh -EncodedCommandover SSH instead of copying a temporary script first. If ms-dev is reachable but Chrome/Edge CDP is not already listening,ensureAppHost: trueorAPP_AUTOMATION_MSDEV_ENSURE_APPHOST=1starts or reuses a dedicated non-disruptive Edge/Chrome apphost on the requested remote-debugging port using a cloned browser-profile cache. By default the refresh script also records the CDP tabs it creates and runs FIFO tab cleanup (APP_AUTOMATION_MSDEV_TAB_GC,APP_AUTOMATION_MSDEV_TAB_GC_KEEP_TICKS, default 2) so recurring loops keep only recent refresh ticks' tabs and never close unrelated user tabs. Failure manifests and refresh headers classify common bridge errors such as connection timeouts and summarize snapshot/skipped-write outcomes without persisting raw command output. When Slack Web is unauthenticated but Slack Desktop is running, the bridge can fall back to the desktop window badge count and writes only a generic unread-count row.app_automation_plan— return the deterministic action plan for an app/action without executing browser automation.app_automation_run— dry-run or execute deterministic allowlisted runner steps such ascli.exec,tendril.run, andsnapshot.write.app_automation_open_bundle_run_once— open Slack, Calendar, Outlook mail/calendar, and Teams browser surfaces once to warm authenticated sessions; passdryRunto inspect the planned browser actions first.app_automation_refresh_start/app_automation_refresh_bundle_start/app_automation_refresh_bundle_run_once/app_automation_refresh_staleness/app_automation_refresh_stale_run_once/app_automation_refresh_status/app_automation_refresh_stop— run non-overlapping Pi-session-local periodic app actions for snapshot refreshes, including standard Slack/Calendar/Outlook/Teams bundle start, one-shot, per-action staleness, stale-only refresh paths, and refresh status with consecutive-error and auth-required diagnostic tracking; one-shot bundles also supportdryRun.app_automation_snapshots_list/app_automation_snapshots_digest/app_automation_snapshot_links/app_automation_snapshots_staleness/app_automation_snapshots_cleanup_plan/app_automation_snapshot_read— inspect, summarize, queryable timestamped/freshness-aware link-list with explicit kind filtering and aliases, optional sorting, matched/truncated counts, freshness/app/kind/host/source/from counts, source context, and clear empty-filter output, freshness-check, and cleanup-plan persisted Slack, Outlook, Teams, calendar, and canvas snapshot artifacts without ad-hoc filesystem reads.app_automation_status— inspect or create the canonical app automation state root./tendril-app [doctor|overview [links] [fresh|stale|unknown] [kind:<kind>] [source:<text>] [from:<text>] [time:<text>] [host:<domain>] [query:<text>|query words] [link-limit:<n>] [link-sort:<order>] [stale-after:<minutes>] [app...]|briefing [stale-after:<minutes>] [app...]|links [[app|all] [fresh|stale|unknown|freshness:<state>] [kind:<kind>] [source:<text>] [from:<text>] [time:<text>] [host:<domain>] [sort:<order>] [limit:<n>] [stale-after:<minutes>] [query]]|staleness|refresh-staleness|bundle|open-bundle|stale-refresh|app action]— quick command for diagnostics, app/action discovery, stale-aware briefing indexes, snapshot links, freshness, and bundle guidance from the Pi UI; omitting the app selector scans all snapshot apps. Shortcut commands/tendril-app-overview,/tendril-app-briefing,/tendril-app-links,/tendril-app-bundle,/tendril-app-open-bundle,/tendril-app-stale-refresh,/tendril-app-refresh-staleness, and/tendril-app-stalenessforward to the matching subcommands for common work-app checks.
Recommended flow: run app_automation_doctor, then check per-action freshness with app_automation_refresh_staleness; on ms-dev setups run app_automation_msdev_cdp_refresh when snapshots are stale or missing, using ensureAppHost for the dedicated Edge/Chrome apphost when needed and leaving FIFO tab cleanup enabled so only refresh-created tabs are closed. Use app_automation_personal_status for the separate personal Google CLI / ~/org/todo.org readiness check. Then run app_automation_work_briefing or /tendril-app briefing for natural-language work-app questions and daily briefing drafts, followed by app_automation_overview (optionally with includeLinks or /tendril-app overview links fresh kind:events host:meet.google.com source:calendar standup link-sort:newest; overview and snapshot-link sections render matched/truncated/scanned counts plus host/source/from/app/kind/freshness distributions and query/kind/freshness/sort including host/source/from/time sort across app samples). For an operator-facing daily Markdown brief, copy the bounded work-briefing samples, overview health lines, focused sanitized links, and personal-status readiness into a short digest; include auth-required/stale-preserved/filtered-empty/ms-dev failure caveats rather than dumping raw app data. Dry-run app_automation_open_bundle_run_once before warming browser sessions; use app_automation_refresh_stale_run_once for normal per-action stale/partial/missing refreshes, and app_automation_refresh_bundle_run_once when a full refresh is explicitly needed; inspect results with the snapshot digest/read/staleness tools. Snapshot digests include link counts when Slack, Calendar, Outlook, or Teams artifacts contain actionable links. Generic snapshots preserve compact source/from/time metadata when extractors provide it, use extracted page titles or sanitized page hostnames as fallback source context, and link listings render each sanitized URL host, query and filter that context plus sanitized link hostnames with service aliases such as meet, gcal, teams, outlook, owa, m365, slack, zoom, and github, while ignoring boolean Slack flags as source noise. Calendar briefing samples stay focused on current-day rows when any are present, instead of filling the sample with older events.
Initial blessed configs cover Slack web notifications, Markdown-to-canvas sync, generic Calendar web events, Outlook web open/snapshot actions, and Teams web open/snapshot actions. Slack/Calendar/Outlook/Teams snapshots preserve safe channel, DM, meeting, and message links while stripping query strings and fragments before writing artifacts, including bounded ancestor-container link scans in the ms-dev extractor; Outlook mail snapshots suppress aggregate mail-list chrome rows; link filters accept natural kind aliases such as events, notifications, mail, chat, mentions, and meetings; Slack notification rows also provide explicit source labels for link context, Teams notification handling keeps the safe badge-count row while suppressing context-menu/accessibility chrome, and Calendar/Microsoft DOM extractors preserve visible time plus bounded Outlook Mail, Outlook Calendar, Teams, Teams Calendar, sender, organizer, and author metadata when available. Auth-required diagnostics apply the same URL redaction to persisted args and stdout/stderr. JSON app configs can also be loaded from APP_AUTOMATION_CONFIG_DIR (default ~/.config/agent-utils/app-automation/apps.d). Slack notification snapshots can already normalize supplied extraction text/JSON into canonical JSON and Markdown artifacts, canvas sync can export Markdown into canonical Markdown/HTML/paste artifacts with sync metadata and prepare live browser replacement steps, and Outlook/Teams include conservative notification and calendar snapshot examples. Executed actions also write safe latest-run.json manifests for durable refresh status without command stdout/stderr. The runner includes a deterministic Playwright bridge for browser.open and dom.extract steps, while app-specific live selectors continue to land incrementally. Explicitly allowlisted low-level runner steps can also be executed through app_automation_run. See docs/app-automation.md for the architecture, snapshot locations, auth policy, periodic refresh shape, and follow-up bead stack.
skill-server (skill-search)
skill-server is a Rust utility that lets agents discover configured skill files and host MCP stdio servers without reloading. It supports a skill-search CLI shorthand and an MCP stdio server built on the same mcp-cli plumbing used by Tendril.
cargo build -p skill-server
skill-search --help
skill-search list --json
skill-search web query latest Rust MCP crate
skill-search mcp stdio
Configuration defaults to .config/ss/config.yaml and can be overridden with SS_CONFIG or --config. See docs/skill-server/README.md for the config schema, MCP tools, and build/test commands.
Firecracker VM Pi extension
The Firecracker VM extension is loaded from extensions/firecracker-vm.js. It gives Pi agents a first-class control plane for microVM workloads that should be visible and controllable by Tendril agents. Firecracker itself is headless, so the extension tracks both serial-console output and any declared host-visible screen/control services such as VNC, noVNC, browser debug endpoints, or web apps.
Available tools:
firecracker_vm_start— create a VM workspace, writefirecracker-config.jsonandtendril-firecracker-manifest.json, and optionally spawnfirecracker --api-sock ... --config-file .... UsedryRun: trueon hosts without Firecracker/KVM to generate the config and manifest only.firecracker_vm_status/firecracker_vm_list— list tracked VMs, lifecycle state, process liveness, manifest paths, service endpoints, logs, and sockets.firecracker_vm_screen— return the serial-console tail plus declared graphical service endpoints that Tendril/browser automation can open or capture.firecracker_vm_stop— stop a tracked VM with SIGTERM followed by bounded SIGKILL and persist lifecycle metadata./firecracker-vms— operator command that summarizes tracked VMs in the TUI.
Key capabilities:
- Configurable VM boot inputs: kernel, optional initrd, rootfs, CPU/memory sizing, kernel args, rootfs read-only mode, logging, metrics, and optional TAP networking.
- Tendril manifest generation with lifecycle metadata, API socket path, serial-console log path, and declared screen/control service URLs.
- Browser/workload configuration via
services, e.g. a noVNC endpoint with{ "name": "novnc", "protocol": "http", "hostPort": 6080, "path": "/vnc.html", "screen": true }. - Visible output access through
firecracker_vm_screen: serial console for all VMs, plus VNC/noVNC/browser URLs for graphical guests. - Lifecycle tracking inside the Pi session with autostop on session shutdown by default.
Example dry-run config/manifest generation:
{
"id": "browser-vm",
"dryRun": true,
"kernelPath": "./vm/vmlinux",
"rootfsPath": "./vm/rootfs.ext4",
"cpuCount": 2,
"memMiB": 2048,
"tapName": "tap0",
"services": [
{ "name": "novnc", "protocol": "http", "hostPort": 6080, "guestPort": 6080, "path": "/vnc.html", "screen": true },
{ "name": "browser-debug", "protocol": "http", "hostPort": 9222, "guestPort": 9222 }
]
}
Tendril share Pi extension
The Tendril share extension is loaded from extensions/tendril-share.js. It gives the user a slash-command path for showing the model a current screen/window, complementing the agent-facing Tendril tools that let the model inspect or control UI.
Available commands:
tendril_settings— native Pi tool that reports the configured Tendril command,--remotetarget,--wsl-tunnelflag, and argument prefix used by alltendril_*tools.tendril_bridge_doctor— native Pi tool that reports the configured Tendril command,--remotetarget,--wsl-tunnelflag, optionally probestendril list --jsonthrough that bridge, and hints when a remote Tendril binary appears too old to accept--wsl-tunnel.tendril_list— native Pi tool equivalent to/tendril list, returning readable display/window targets plus structured target data.tendril_capture— native Pi tool equivalent to/tendril window|display, capturing a target by id or unique name/title/app substring and returning PNG image content directly to the model by default. It also includes the currenttendril listtarget context by default; setpathOnly: trueto omit image content andincludeList: falseto omit target-list context. Passremote: "<host>"(and optionallywslTunnel: true|false) to capture from a Tendril bridge on a specific machine for that call only;remote: ""forces the local bridge even whenAGENT_UTILS_TENDRIL_REMOTEis set.tendril_describe— native Pi tool equivalent to the model-facing half of/tendril describe, returning the captured PNG plus an objective visual-description prompt so the calling model can inspect the image directly. It supports the samepathOnlyandincludeListflags.tendril_stream— native Pi tool equivalent to/tendril stream start|status|stop; started streams queue low-resolution follow-up screenshot messages for the model by default, update the kitty terminal preview with a stable stream image id when preview is enabled, and the first frame includes currenttendril listtarget context. UsepathOnly: trueorincludeList: falseto reduce payload/context./tendril list— runstendril list --jsonand shows readable display/window targets./tendril window <id-or-name> [prompt]— captures a Tendril window to PNG and sends a user message containing the screenshot image plus optional prompt text./tendril display <id-or-name> [prompt]— captures a Tendril display to PNG and sends it to the model./tendril screen <id-or-name> [prompt]— alias for/tendril display./tendril describe window <id-or-name> [prompt]//tendril describe display <id-or-name> [prompt]— captures the target, describes it with a configurable image-capable model, and sends the textual description to the active model./tendril-describe window <id-or-name> [prompt]//tendril-describe display <id-or-name> [prompt]— shortcut aliases for the describe path./tendril stream window <id-or-name> [seconds] [prompt]//tendril stream display <id-or-name> [seconds] [prompt]— starts low-resolution periodic screenshot sharing and refreshes the kitty preview widget with the latest frame using a stable stream image id. Defaults to 30 seconds, clamps to at least 10 seconds, and captures at 640×360./tendril stream status//tendril stream stop— inspect or stop the active screenshot stream./tendril settings//tendril-settings— open an interactive overlay for Tendril settings such as the description model and kitty preview toggle; changes are persisted to settings.json and are read by later captures/descriptions without restart.- Slash capture/stream flags:
--path-onlyomits image content and sends just text/path context;--no-listomits the default target-list context.
Set AGENT_UTILS_TENDRIL_REMOTE=<host> to wrap Tendril invocations with tendril --remote <host>, and set AGENT_UTILS_TENDRIL_WSL_TUNNEL=1 to add Tendril's --wsl-tunnel hop for WSL-to-Windows-host control (for example ms-dev with a visible Windows tendril.exe). These env vars set the session-wide default bridge; every tendril_* tool (tendril_list, tendril_capture, tendril_describe, tendril_stream, tendril_bridge_doctor) and the /tendril slash command additionally accept a per-call remote host override (and optional wslTunnel toggle) so a single Pi session can transparently capture applications across different machines without changing the env. In the slash command use --remote <host> / --remote=<host>, --local (or --no-remote) to force the local bridge, and --wsl-tunnel / --no-wsl-tunnel to toggle the tunnel hop for that command. As tool params, pass remote: "<host>" and wslTunnel: true|false; an empty remote: "" forces the local bridge even when AGENT_UTILS_TENDRIL_REMOTE is set, and omitted keys fall back to the env defaults. Capture targets can be exact ids or unique case-insensitive substrings from Tendril target titles, names, or app names (for example /tendril window Safari or tendril_capture({ kind: "window", target: "Safari" })). Captures are written under the active Pi session directory when available, or a process-scoped temp folder otherwise, and the extension records a visible history message with the saved file path for slash-command captures. Capture/share and first stream-frame messages include the current readable tendril list output by default so the model can ask for a follow-up crop/window without first listing targets; opt out with --no-list or includeList: false. If the agent is already streaming, screenshot messages are queued as follow-ups so the user can share visual context without interrupting the current turn. Description commands default to github-copilot/claude-opus-4.8 (falling back through known Copilot/internal Opus 4.8/4.7 and LiteLLM Opus ids if the default is not registered), so Pi's GitHub Copilot provider can use its normal ~/.config/gh-auth-tokens auth path; persist an override with tendril.describeModel or agentUtils.tendril.describeModel in settings.json, or override at runtime with TENDRIL_SHARE_DESCRIBE_MODEL=provider/model. The tendril_describe native tool instead returns image content to the calling model directly, avoiding a second VLM call. When a capture comes from a --remote <host> bridge, the source machine is associated with the result: it is added to the description prompt sent to the inference model, surfaced in the shared description text (source machine <host> / Source machine: <host>), recorded in capture-history metadata, and namespaced into the saved screenshot filename so multiple concurrent remote captures of the same target from different machines never collide.
Kitty image preview Pi extension
The kitty image preview extension is loaded from extensions/kitty-image-preview.js and uses shared protocol helpers in extensions/kitty-graphics.js. It is a first-class Pi package extension like search_web: install this repo as a Pi package, then the tools become available to the agent without shelling out to kitty icat.
Available tools:
kitty_image_preview_add— add a PNG/APNG image and optionally show it immediately.kitty_image_preview_capture— capture a Tendril screenshot into the current Pi session screenshot folder and show it immediately.kitty_image_preview_add_folder— add a sorted image series from a directory, optionally recursively.kitty_image_preview_show— navigatecurrent,next,previous,first,last,index,hide, orclear.kitty_image_preview_animate— start or stop lightweight frame animation by cycling a loaded image series.kitty_image_preview_cycle— start (action: "start", intervalSeconds: 5) or stop (action: "stop") timed cycling through the loaded gallery. Slash-command equivalents:/image-next,/image-prev,/image-hide,/image-show,/image-clear,/image-config [key=value …],/image-start-cycle [seconds],/image-stop-cycle.kitty_image_preview_stream_start/kitty_image_preview_stream_stop/kitty_image_preview_stream_status— show an ephemeral Tendril screenshot stream using a two-file temp buffer so frames do not accumulate on disk or in model context. SetintervalMs: 0for max non-overlapping Tendril capture rate.kitty_image_preview_stream_sample— persist one selected stream frame, optionally withdescribe: true.kitty_image_preview_playwright_start— watch a PNG path written by Playwrightpage.screenshot()calls so users see a live browser mirror while the agent can keep using DOM-only context.kitty_image_preview_status— inspect loaded images, active index, transfer mode, and passthrough detection.
Key capabilities:
- Native kitty graphics APC serialization with chunking, PNG file or in-memory transfer, and tmux DCS passthrough autodetection.
- Unicode placeholder placement under tmux so the image is anchored to the widget text cells and scrolls with the pane instead of floating at the outer terminal cursor. In this mode the inline preview is framed as a horizontal rule, a
name (current/total)header line, the image, and a closing horizontal rule (the controls hint moves to the status line). - Transmit-once / placement-only redraws in Unicode-placeholder mode: the base64 PNG payload is uploaded only once per
(image id + render signature)and subsequent repaints emit placement-only frames (the placeholder cells carry the image), which avoids re-sending image bytes over SSH/tmux on every redraw. A new image, geometry/transport/config change, or a terminal reattach (session_start) re-transmits so a freshly attached kitty client holds the payload. This widget-boundary behavior is guarded bytest/kitty-image-preview-widget.test.jsand a runnable harness,npm run pi-graphics:widget-smoke(scripts/test-kitty-image-preview-widget-smoke.mjs), which drivesKittyImagePreviewWidget.rendertwice and assertsfirstUpload=true,redrawUpload=false, andreattachReupload=truewithout needing a live terminal. - First-party screenshot capture via
tendril capture --output, saved under a per-sessionkitty-image-preview-screenshotsfolder by default. - Persistent Pi widget mounted above or below the editor with configurable cell width/height and captioning.
- Automatic screenshot-friendly placement via
placement: "auto"(the default): outside tmux on wide terminals it uses a right-side side panel sized to the current image, capped by 50% of terminal width and the visible height above the input box, so chat text reflows beside it; inside tmux or on narrow terminals it falls back to the inline above-editor widget. - Negative z-index rendering by default for direct cursor placement so images can sit underneath text;
background: trueuses an extra-low z-index for background-style placement. In tmux placeholder mode, image stacking follows kitty's placeholder rendering semantics. - Alpha/transparency support through PNG/APNG and kitty's compositor.
- Lightweight animation support by cycling folder/series frames at configurable intervals.
- Optional
describe: trueon still-image tools to send just that image to a VLM for an objective visual description. Screenshot descriptions use a separate full-resolution Tendril capture even when the terminal preview is downscaled. Defaults tolitellm-anthropic/claude-opus-4-7, override withdescribeModelorKITTY_IMAGE_PREVIEW_DESCRIBE_MODEL. - Optional stream descriptions with
describe: trueordescribeIntervalSecs, recorded as text metadata only. Stream previews can stay low-res while description frames are captured separately at full resolution in the background. - Playwright visual mirroring can automatically run
playwright-cli screenshot --filename <temp>on an interval for the active/session browser, or can watch a screenshot path written by external Playwright code whenautoScreenshot: false. Frames remain display-only unless sampled. - Session-state reconstruction from prior tool results so loaded image lists survive Pi session reloads.
- Scoped kitty image cleanup: the extension tracks the kitty graphics image ids it transmits and only ever issues per-image deletes (
d=i,i=<id>) for those owned ids when hiding, clearing, or shutting down. It never emits a global delete-all (d=A) sequence, so running it inside another kitty graphics consumer (e.g. caco) does not erase unrelated images or the surrounding UI. - User-facing TUI controls: the preview widget/status line advertises
/image-prev,/image-next,/image-hide,/image-show, and/image-clearso the operator can control multi-image galleries without waiting for an agent tool call./image-config [key=value …]shows or updates render params (placement, placementMode, transferMode, passthrough, zIndex, columns, rows, maxRows, minRows, background, showCaption, clearPrevious) at runtime;/image-start-cycle [seconds]and/image-stop-cyclecontrol cycling. (The legacy/kitty-*slash-command aliases were removed; thekitty_image_preview_*tool names are unchanged.)
Example image tool use:
{
"path": "./artifacts/preview.png",
"config": {
"columns": 48,
"placement": "auto",
"transferMode": "auto",
"passthrough": "auto",
"placementMode": "auto",
"zIndex": 0
}
}
Example screenshot capture tool use:
{
"targetKind": "display",
"maxWidth": 1200,
"config": {
"columns": 48,
"placement": "auto"
}
}
Example fixed right-side screenshot preview:
{
"targetKind": "display",
"maxWidth": 1200,
"config": {
"columns": 48,
"placement": "rightOverlay",
"transferMode": "auto"
}
}
The native protocol path currently accepts PNG/APNG input. Convert JPEG/WebP/GIF assets to PNG first when using the widget directly. placement: "auto" chooses the fixed right-side panel only when it should be ergonomic; use "rightOverlay", "aboveEditor", or "belowEditor" to force a location. placementMode: "auto" uses anchored Unicode placeholders by default so previews update in-place without moving the terminal cursor or flooding scrollback; use "cursor" only for debugging terminal-specific behavior. The right-side panel dynamically fits the image to the available frame, clamps total reserved width (including left padding) to 50% of the terminal, never exceeds the visible height above the editor/input area, and bottom-aligns the image immediately above that input area. If tmux passthrough or an older Pi runtime prevents side-panel rendering, it falls back to the inline above-editor widget. Preview images stay out of model context unless describe: true is explicitly requested for a still image.
Android CLI / emulator Pi extension
The Android extension is loaded from extensions/android.js. It wraps Google's Android command-line tools (adb, emulator, sdkmanager) so the model can diagnose/install the Android SDK and capture emulator/device screens.
Available tools:
android_cli_doctor— diagnoses Android CLI /adb/emulator/ SDK-manager availability and reports the relevant install/update commands.android_cli_help— prints concise helper usage for install, update, screenshots, and streams.android_cli_install— installs the Android CLI via Google's install script. Requiresconfirmed: true; otherwise returns the exact command as dry-run guidance.android_cli_update— updates Android CLI / SDK packages (android updateorsdkmanager --update). Requiresconfirmed: true.android_emulator_screenshot— captures anadbscreenshot as PNG, returns image content for immediate Pi display, and saves the file forkitty_image_preview_add.android_emulator_stream— bounded repeatedadbscreencaps for a short live emulator screenshot stream.
The SDK location is resolved from ANDROID_HOME / ANDROID_SDK_ROOT. Installs use https://dl.google.com/android/cli/latest/linux_x86_64/install.sh.
Xvfb virtual display Pi extension
The Xvfb extension is loaded from extensions/xvfb.js. It is the opt-in orchestration follow-up to the headless display-detection helper in extensions/lib/headless-display.js: on a headless Linux node (no DISPLAY/WAYLAND_DISPLAY, not WSLg) it can spawn an Xvfb virtual display and export DISPLAY so display-dependent tools — Tendril capture, Android screenshots, app-automation browser — can run instead of dead-ending.
It never spawns unprompted: the model invokes xvfb_ensure explicitly, and the tool refuses unless the node is genuinely headless (override with force: true), refuses if the Xvfb binary is not on PATH, and selects a unique display number by probing /tmp/.X11-unix so it does not collide with another agent's display on the same host. The spawned process is tracked and torn down on session shutdown (or via xvfb_stop), mirroring the Firecracker extension's autostop lifecycle.
xvfb_ensure— opt-in: spawn an Xvfb virtual display and exportDISPLAY. Idempotent (reuses a live session display); supportsscreengeometry (default1920x1080x24), a customcommand/path, anddryRun: trueto report the spawn plan (binary, args, chosenDISPLAY) without spawning.xvfb_stop— stop the session-owned Xvfb display (if any) and unset the exportedDISPLAY, withforceAfterMscontrolling the SIGTERM→SIGKILL grace period.xvfb_status— report current display availability (reusing the detection helper's summary) and whether a session-owned virtual display is active.
The policy, free-display selection, and command construction live in extensions/lib/xvfb.js and are pure over injected probes, so they are unit-tested without spawning or touching the real filesystem.
Web search Pi extension
The web-search extension is loaded from extensions/web-search.js. It registers the search_web native tool, which answers a query with live web results grounded through GitHub Copilot's Responses API and returns a cited answer.
Configuration (environment):
WEB_SEARCH_COPILOT_TOKEN_FILE— explicit path to a Copilot bearer token file. When set (or pointed at a non-default path), it takes precedence. Otherwise the bearer is read from Pi's auto-refreshedauth.json(see below), falling back to~/.config/gh-auth-tokens/copilot.token.WEB_SEARCH_COPILOT_AUTH_JSON— path to Pi'sauth.jsonholding the auto-refreshed Copilot bearer (default~/.pi/agent/auth.json). The bearer is taken from<key>.access; Pi keeps this fresh, avoiding the stale static-token-file problem.WEB_SEARCH_COPILOT_AUTH_JSON_KEY— top-level key inauth.jsonwhose.accessfield holds the bearer (defaultgithub-copilot).WEB_SEARCH_MODEL— upstream Copilot model (defaultgpt-5.3-codex).WEB_SEARCH_MAX_OUTPUT_TOKENS— maximum output tokens (default16000).WEB_SEARCH_COPILOT_API_BASE— API base URL (defaulthttps://api.githubcopilot.com/v1).WEB_SEARCH_EDITOR_VERSION—editor-versionrequest header (defaultvscode/1.103.1).
A separate Python MCP implementation also lives under web-search/; this extension is the in-process Pi tool wrapper.
Compaction continue guard Pi extension
The compaction-continue-guard extension is loaded from extensions/compaction-continue-guard.js. After a compaction, Pi core cannot continue an agent loop when the rebuilt transcript ends on an assistant message (it throws Cannot continue from message role: assistant). A compaction can legally retain a suffix whose newest message is assistant-authored, so this extension appends a single hidden custom/user-role checkpoint entry immediately after compaction. A later automatic retry or manual continue then starts from a provider-valid user/custom role instead of failing.
The guard is enabled by default; disable it by setting PI_COMPACTION_CONTINUE_GUARD to a falsey value (0, false, off, no, or disabled).
Tendril command helper
extensions/tendril-command.js is a shared helper module (not a tool/command surface of its own) that builds the Tendril bridge invocation used by all tendril_* tools and /tendril commands. It resolves the Tendril binary and bridge options from the environment:
AGENT_UTILS_TENDRIL_COMMAND/TENDRIL_COMMAND— Tendril binary to invoke (defaulttendril).AGENT_UTILS_TENDRIL_REMOTE/TENDRIL_REMOTE— wrap calls with--remote <host>.AGENT_UTILS_TENDRIL_WSL_TUNNEL/TENDRIL_WSL_TUNNEL— add Tendril's--wsl-tunnelhop (truthy:1,true,yes, oron).
See the Tendril share Pi extension section above for the user-facing commands and tools that consume this helper.
GitHub Pages tool inventory
This repository includes a minimal GitHub Pages site in docs/ with a concise inventory of the Cacophony, Pi, UI automation, and repo-local tools available to agents/operators.
Publishing path:
- GitHub Pages source: Deploy from a branch
- Branch:
main - Folder:
/docs
Update and validate the rendered page with:
npm run docs:build
npm run docs:check
Preview locally with:
npm run docs:build
python3 -m http.server --directory docs 8000
Then open http://localhost:8000/.