convez.ai
MCP ServerIntegração & APIsIntermediário
0xMassi

Webclaw

0xMassi/webclaw

Extrai conteúdo web limpo para agentes IA e pipelines RAG

Ver no GitHub
Muito bomPluginScore · de 0 a 100

MCP server para extração rápida e local-first de conteúdo web em markdown, JSON e formatos prontos para LLMs. Oferece CLI, REST API e integração nativa com Claude, Cursor e outras ferramentas MCP. Resolve o problema fundamental do web scraping: converter URLs em conteúdo limpo e estruturado, sem HTML sujo, scripts ou boilerplate.

Instalação

$npx convez-ai 0xMassi/webclaw

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
2,5 mil
Forks
247
Acompanhando
13
Saúde
71%
Último commit
há 1 dia
Versão
v0.7.4
Issues abertas
6
Contribuidores
6

Casos de uso

  • Alimentar agentes IA com conteúdo web estruturado via MCP
  • Extrair dados de websites para RAG pipelines e busca semântica
  • Automatizar web scraping sem lidar com HTML sujo ou bloqueios

Veredito

Ferramenta sólida de extração web com forte foco em integração MCP. Resolve um problema real para agentes IA que precisam consumir conteúdo web, oferecendo saídas limpas e otimizadas para LLMs. Vale a pena para quem trabalha com RAG ou agents que lidam com web scraping.

README

A tradução para o português deste README ainda está na fila.


Most web scraping tools give your agent one of two bad outputs:

  • a blocked page, login wall, or empty app shell
  • raw HTML full of nav, scripts, styling, ads, and duplicated boilerplate

webclaw.io is the hosted web extraction API for webclaw. This repo contains the open-source CLI, MCP server, extraction engine, and self-hostable server.

webclaw turns a URL into clean content your tools can actually use.

webclaw https://example.com --format markdown
# Example Domain

This domain is for use in illustrative examples in documents.

You may use this domain in literature without prior coordination or asking for permission.

Use it from the terminal, wire it into Claude/Cursor through MCP, call the hosted API from your app, or self-host the OSS server.


Install

Agent setup

The fastest way to connect webclaw to Claude Code, Claude Desktop, Cursor, Windsurf, OpenCode, Codex CLI, and other MCP-compatible tools:

npx create-webclaw

The installer detects supported clients and configures the MCP server for you.

Homebrew

brew tap 0xMassi/webclaw
brew install webclaw

Prebuilt binaries

Download macOS, Linux, and Windows binaries from GitHub Releases.

Docker

docker run --rm ghcr.io/0xmassi/webclaw https://example.com

Cargo

cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli
cargo install --git https://github.com/0xMassi/webclaw.git webclaw-mcp

If building from source fails because native build tools are missing, install the platform prerequisites:

OSCommand
Debian / Ubuntusudo apt install -y pkg-config libssl-dev cmake clang git build-essential
Fedora / RHELsudo dnf install -y pkg-config openssl-devel cmake clang git make gcc
Archsudo pacman -S pkg-config openssl cmake clang git base-devel
macOSxcode-select --install

Quick Start

Scrape one page

webclaw https://stripe.com --format markdown

Return LLM-optimized text

webclaw https://docs.anthropic.com --format llm

Keep only the main content

webclaw https://example.com/blog/post --only-main-content

Include or exclude selectors

webclaw https://example.com \
  --include "article, main, .content" \
  --exclude "nav, footer, .sidebar, .ad"

Crawl a documentation site

webclaw https://docs.rust-lang.org --crawl --depth 2 --max-pages 50

Workflow examples

Extract brand assets

webclaw https://github.com --brand

Compare a page over time

webclaw https://example.com/pricing --format json > pricing-old.json
webclaw https://example.com/pricing --diff-with pricing-old.json

MCP Server

webclaw ships with an MCP server for AI agents.

npx create-webclaw

Manual config:

{
  "mcpServers": {
    "webclaw": {
      "command": "~/.webclaw/webclaw-mcp"
    }
  }
}

Then ask your agent things like:

Scrape these competitor pricing pages and summarize the differences.
Crawl this documentation site and prepare clean context for a RAG index.
Extract the brand colors, fonts, and logos from this company website.

Tools

ToolWhat it doesLocal
scrapeExtract one URL as markdown, text, JSON, LLM format, or HTMLYes
crawlFollow same-origin links and extract discovered pagesYes
mapDiscover URLs without extracting every pageYes
batchScrape multiple URLs in parallelYes
extractConvert page content into structured dataYes, with local or configured LLM
summarizeSummarize a pageYes, with local or configured LLM
diffCompare page content snapshotsYes
brandExtract colors, fonts, logos, and metadataYes
searchSearch the web and scrape resultsHosted API
researchMulti-source research workflowHosted API

SDKs

npm install @webclaw/sdk
pip install webclaw
go get github.com/0xMassi/webclaw-go
import { Webclaw } from "@webclaw/sdk";

const client = new Webclaw({ apiKey: process.env.WEBCLAW_API_KEY! });

const page = await client.scrape({
  url: "https://example.com",
  formats: ["markdown"],
  only_main_content: true,
});

console.log(page.markdown);
from webclaw import Webclaw

client = Webclaw(api_key="wc_your_key")

page = client.scrape(
    "https://example.com",
    formats=["markdown"],
    only_main_content=True,
)

print(page.markdown)
curl -X POST https://api.webclaw.io/v1/scrape \
  -H "Authorization: Bearer $WEBCLAW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown"],
    "only_main_content": true
  }'

Output Formats

FormatUse it when you need
markdownClean page content with structure preserved
llmCompact context for agents and RAG pipelines
textPlain text with minimal formatting
jsonStructured metadata, links, images, and extracted fields
htmlCleaned HTML for custom processing

Local First, Hosted When Needed

The CLI and MCP server work locally without an account for the core extraction path.

Use the hosted API at webclaw.io when you need:

  • protected-site access without managing infrastructure
  • JavaScript rendering
  • async crawl and research jobs
  • web search
  • watches and production usage tracking
  • SDKs for application code
export WEBCLAW_API_KEY=wc_your_key

webclaw https://example.com --cloud

What You Can Build

Use caseExample
AI agent web accessGive Claude, Cursor, or another MCP client clean page context
RAG ingestionCrawl docs, help centers, blogs, and knowledge bases
Competitor monitoringTrack pricing pages, changelogs, docs, and product pages
Structured extractionTurn messy pages into typed JSON for automations
Research workflowsSearch, scrape, summarize, and cite multiple sources
Brand intelligenceExtract logos, colors, fonts, and social metadata

Architecture

webclaw/
  crates/
    webclaw-core     HTML to markdown, text, JSON, and LLM-ready output
    webclaw-fetch    Fetching, crawling, batching, and mapping
    webclaw-llm      Local and hosted LLM provider support
    webclaw-pdf      PDF text extraction
    webclaw-mcp      MCP server for AI agents
    webclaw-cli      Command-line interface

webclaw-core is pure extraction logic: no network I/O, small surface area, and usable independently from the fetching layer.


Configuration

VariableDescription
WEBCLAW_API_KEYHosted API key
OLLAMA_HOSTOllama URL for local LLM features
OPENAI_API_KEYOpenAI-compatible LLM provider key
OPENAI_BASE_URLOpenAI-compatible base URL
ANTHROPIC_API_KEYAnthropic-compatible LLM provider key
ANTHROPIC_BASE_URLAnthropic-compatible base URL
WEBCLAW_PROXYSingle proxy URL
WEBCLAW_PROXY_FILEProxy pool file

Contributing

The most useful contributions right now are practical and small:

  • add examples for real agent and RAG workflows
  • improve SDK snippets
  • report pages that extract poorly
  • add failing fixtures for messy HTML
  • improve docs for MCP clients and local setup
  • test the CLI on more Linux/macOS environments

Good first places to start:

If a page extracts badly, include:

URL:
Command or API request:
Expected output:
Actual output:
Format used: markdown / llm / text / json / html
CLI, MCP, SDK, or API:

Please remove secrets, cookies, private tokens, and customer data from logs before posting.


Infrastructure Partner


Studio Partners


Community Plugins

Third-party plugins that integrate webclaw with AI agent platforms:

PluginPlatformWhat it does
openclaw-webclawOpenClawNative webclaw v1 API plugin with 9 tools: scrape, search, crawl, extract, summarize, diff, map, batch, brand
hermes-webclawHermes AgentWeb search provider and 9 dedicated tools for the full v1 API surface. Install with hermes plugins install jal-co/hermes-webclaw

Built a webclaw integration? Open a PR to add it here.


Contributors

Thanks to everyone improving webclaw through issues, examples, docs, bug reports, and pull requests.


Star History


License

AGPL-3.0

Ver o original no GitHub

Bom para

Relacionados