convez.ai
MCP ServerIntegração & APIsIntermediário
lukaszraczylo

ReadItAll

lukaszraczylo/mcp-readitall

Converte websites em Markdown com renderização real de browser

Ver no GitHub
InicialPluginScore · de 0 a 100

Servidor MCP em Go que converte qualquer website em Markdown limpo, incluindo SPAs modernas que precisam de navegador real. Suporta sessões autenticadas por host, permitindo acessar conteúdo protegido por login.

Instalação

$npx convez-ai lukaszraczylo/mcp-readitall

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
1
Forks
0
Acompanhando
0
Saúde
28%
Último commit
há 2 dias
Versão
v0.1.74
Issues abertas
0
Contribuidores
1

Casos de uso

  • Extrair conteúdo de websites dinâmicos para processamento por IA
  • Acessar páginas protegidas por autenticação de forma automatizada
  • Converter artigos e documentos web em Markdown para arquivamento
  • Extrair seções específicas de páginas usando seletores CSS

Veredito

Ferramenta sólida para integração com Claude e clientes MCP. Resolve problema real de conversão de conteúdo web dinâmico para texto estruturado, com suporte a autenticação — útil para workflows que precisam ingerir dados de websites protegidos.

README

A tradução para o português deste README ainda está na fila.

mcp-readitall

A Go-based Model Context Protocol (MCP) server that fetches any website — including modern SPAs that need a real browser to render — and converts the page into clean Markdown. It also supports per-host authenticated sessions so you can read content that lives behind a login wall.

The server speaks MCP over stdio and is designed to be wired into any MCP-aware client (Claude Desktop, Claude Code, Cursor, etc.).

Features

  • Real browser rendering via chromedp (a pure-Go Chrome DevTools Protocol client) + system Chromium — handles SPAs, JS hydration, lazy-loaded content, client-side routing. No Node.js bridge.
  • Markdown conversion with the html-to-markdown plugin pipeline: tables, code blocks (with language detection), strikethrough, commonmark semantics, relative-URL resolution.
  • Per-host authenticated sessions — the login tool opens a visible browser window for you to authenticate, then persists cookies + localStorage so future reads reuse the session automatically.
  • Session lifecycle tools — list_sessions and clear_session for inspection and revocation.
  • Content extraction — extract and the selector parameter on read_url let you pull just the part of a page you care about.
  • Cross-process safe — session state is persisted to disk (0600) so the server can restart without losing auth.

Tools

ToolPurpose
read_urlFetch a URL (with optional CSS selector), render JS, return Markdown.
loginOpen a browser, let the operator log in, persist the session.
list_sessionsShow every host for which a session is saved.
clear_sessionDelete a saved session.
extractRead a URL and return only the part under a CSS selector.

read_url arguments

FieldTypeDescription
urlstring (required)Absolute http(s) URL.
selectorstringOptional CSS selector to extract a region.
wait_forstringOptional CSS selector to wait for (useful for SPAs that hydrate after networkidle).
timeout_secondsintOverall timeout. Default 30.
max_charsintTruncate the returned Markdown. Default ~200k.

login arguments

FieldTypeDescription
urlstring (required)The URL to open (typically the login page).
success_urlstringSubstring of the post-login URL (e.g. /dashboard). Polled every second.
success_selectorstringCSS selector that becomes visible once auth succeeds (e.g. an avatar menu).
timeout_secondsintHow long to wait for the operator to complete login. Default 300.

success_url or success_selector is required so the tool knows when to stop waiting and save the session.

Build & install

Requirements:

  • Go 1.22+
  • A Chromium-family browser on $PATH or installed at the standard location — google-chrome, chromium, chromium-browser, or Edge. macOS: brew install --cask chromium (or the standard Chrome install). Linux: apt install chromium / dnf install chromium.
# Build the server binary
go build -o bin/mcp-readitall ./cmd/readitall

If the project is not under git you may need -buildvcs=false: go build -buildvcs=false -o bin/mcp-readitall ./cmd/readitall

chromedp will auto-discover Chromium via exec.LookPath and the usual Chromium env vars (CHROME_BIN, CHROMIUM_BIN, etc.). If you have a non-standard install, point at the binary directly:

CHROME_BIN=/Applications/Chromium.app/Contents/MacOS/Chromium ./bin/mcp-readitall

Wire into an MCP client

Claude Desktop (~/Library/Application Support/Claude/claude_desktop_config.json)

{
  "mcpServers": {
    "readitall": {
      "command": "/absolute/path/to/mcp-readitall/bin/mcp-readitall"
    }
  }
}

Claude Code (.mcp.json in your project)

{
  "mcpServers": {
    "readitall": {
      "command": "/absolute/path/to/mcp-readitall/bin/mcp-readitall"
    }
  }
}

Restart the client after editing the config.

Usage flow

1. Read a public page

Ask your MCP client:

"Read https://news.ycombinator.com and summarize the top 5 stories"

Under the hood the model calls read_url. The server launches a headless Chromium, navigates, waits for the body to be ready (plus a short grace period for hydration, or a wait_for selector if supplied), converts the HTML to Markdown, and returns it.

2. Read an authenticated page

"Log me into GitHub, then read my notifications"

The model first calls:

{
  "name": "login",
  "arguments": {
    "url": "https://github.com/login",
    "success_url": "/",
    "timeout_seconds": 300
  }
}

A real browser window pops up on your machine. Log in normally. When the URL changes to contain /, the tool saves the session and closes the window. Subsequent calls to read_url for github.com will reuse the saved cookies + localStorage automatically.

To invalidate a session:

"Clear my session for github.com"

The model calls clear_session.

Session storage

Saved sessions live in:

  • ~/.mcp-readitall/sessions/<host>/state.json by default
  • Override with READITALL_SESSIONS_DIR=/some/path

State files are written with mode 0600 (owner read/write only) and use Playwright's standard storage state shape ({cookies:[…], origins:[{origin, localStorage:[…]}]}) — sessions captured by Playwright can be loaded by us, and vice versa. Treat them as secrets — they grant access to your authenticated sessions.

Tests

go test -race ./...

The suite is hermetic — no browser is launched. It covers:

  • Hostname normalization (good / bad / edge).
  • Storage-state validation.
  • Session-store CRUD and concurrency safety.
  • HTML→Markdown conversion (headings, lists, tables, code, entities, URL resolution, security-sensitive stripping of <script> / <style> / comments).
  • Tool helpers (URL host extraction, comma-separated selector parsing, output formatter).

Limitations

  • The login tool needs a visible browser window. Running it on a remote / headless host with no display (and no VNC / X-forwarding) will not work — Chrome exits immediately when it has no window to draw to.
  • Requires a Chromium-family browser on the host. chromedp does not bundle one.
  • The reader is single-tab per call. Reading multiple pages in parallel inside one tool call is not supported yet.
  • We treat networkidle as "body is ready + 500 ms grace" (or whatever you pass in wait_for). Sites that keep firing background requests indefinitely may need a wait_for selector rather than rely on automatic settling.

License

MIT.

Ver o original no GitHub

Bom para

Relacionados