convez.ai
MCP ServerBusca & PesquisaIntermediário
raintree-technology

docpull

raintree-technology/docpull

Converta documentação web pública em Markdown pronto para IA, sem navegador.

Ver no GitHub
BomPluginScore · de 0 a 100

Servidor MCP e ferramenta CLI que converte documentação pública e sites estáticos em Markdown limpo, NDJSON e context packs prontos para LLMs e RAG. Realiza crawling local sem JavaScript, ideal para preparar bases de conhecimento offline e fluxos de agentes de IA.

Instalação

$npx convez-ai raintree-technology/docpull

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
29
Forks
1
Acompanhando
1
Saúde
87%
Último commit
há 5 dias
Versão
v6.10.0
Issues abertas
3
Contribuidores
6

Casos de uso

  • Extrair documentação técnica para RAG e retrieval
  • Preparar bases de conhecimento offline para agentes de IA
  • Converter blogs e referências de API em contexto estruturado para LLMs
  • Indexar sites com busca FTS5 para pesquisa local
  • Gerar context packs portáveis em formato OKF

Veredito

Ferramenta sólida e madura para extrair conhecimento de sites públicos em formato pronto para IA. Abordagem sem JavaScript é honesta sobre limitações, ideal para documentação estática e fluxos RAG em produção.

README

A tradução para o português deste README ainda está na fila.

docpull

Turn public documentation sites into local Markdown, NDJSON, and agent-ready context packs. No browser required.

Python 3.10+ PyPI version PyPI downloads GitHub stars License: MIT

docpull is a Python CLI, SDK, and MCP server that fetches public static or server-rendered web pages and converts them into clean, auditable local artifacts for LLMs, retrieval-augmented generation (RAG), offline research, and agent workflows.

It works best on documentation sites, blogs, API references, OpenAPI specs, vendor pages, and other public pages where the useful content is available in HTML or embedded page data.

docpull is not a browser and does not render JavaScript. JS-only pages are skipped with a clear reason. See Scraping Boundary and Alternatives for the full boundary.

Install

pip install docpull

Install optional extras as needed:

pip install 'docpull[llm]'           # tiktoken for token-accurate chunking
pip install 'docpull[trafilatura]'   # alternative extractor for noisy pages
pip install 'docpull[mcp]'           # stdio MCP server
pip install 'docpull[parallel]'      # Parallel context packs
pip install 'docpull[observability]' # Raindrop benchmark tracing
pip install 'docpull[all]'           # all optional extras

30-Second Usage

docpull https://docs.python.org/3/library/asyncio.html --single -o ./asyncio-docs

Example output:

asyncio-docs/
  index.md
  corpus.manifest.json

Markdown includes source metadata and readable page content:

---
title: "asyncio - Asynchronous I/O"
source: https://docs.python.org/3/library/asyncio.html
source_type: "sphinx"
---

# asyncio - Asynchronous I/O

asyncio is a library to write concurrent code using the async/await syntax.

Stream chunked NDJSON for agents and RAG:

docpull https://docs.python.org/3/library/asyncio.html \
  --single \
  --profile llm \
  --stream | jq .

Each line is a JSON document:

{"schema_version":1,"document_id":"doc_...","chunk_id":"chunk_...","url":"https://docs.python.org/3/library/asyncio.html","title":"asyncio - Asynchronous I/O","content":"asyncio is a library to write concurrent code...","source_type":"sphinx","chunk_index":0,"token_count":842}

Common Workflows

# Crawl a site and write Markdown files
docpull https://docs.example.com -o ./docs-example

# Stream LLM-ready NDJSON chunks from a site
docpull https://docs.example.com --profile llm --stream | jq .

# Write SQLite with an FTS5 search index
docpull https://docs.example.com --format sqlite -o ./docs-db

# Build an Open Knowledge Format (OKF) bundle for portable source packs
docpull https://example.com --profile okf -o ./site-okf

More examples live in CLI Recipes.

Use docpull when you need to:

  • Convert public docs, blogs, API references, vendor pages, and OpenAPI specs into Markdown or chunked NDJSON for LLM and RAG pipelines.
  • Give an agent a local tool for fetching, caching, grepping, and reading web sources.
  • Build repeatable context packs with stable IDs, hashes, manifests, and source metadata.
  • Mirror public web content for offline work while preserving attribution.

Why docpull?

docpull is designed for agent and RAG workflows, not just downloading pages.

Needdocpull gives you
Clean MarkdownArticle-focused extraction with source metadata
LLM chunksNDJSON streaming and optional token-aware chunking
RepeatabilityStable document IDs, chunk IDs, hashes, and manifests
Offline workCached archives and mirrored source artifacts
Agent accessLocal CLI, Python SDK, and stdio MCP server
Safer fetchingHTTPS defaults, robots.txt compliance, SSRF protections, and redirect guards

Supported Sources

docpull uses async HTTP instead of browser automation and includes special handling for common documentation and API surfaces.

Source shapeSupport
Static HTML / SSR docsExtracts article or document regions
Next.js / MintlifyParses static HTML and __NEXT_DATA__ when available
OpenAPI / SwaggerRenders specs into Markdown
Docusaurus / Sphinx / MkDocsExtracts static article or document regions
VitePress / VuePress / Astro StarlightExtracts static docs content
GitBook / ReadMe.ioExtracts available article or content regions
Redoc / ScalarExtracts static API reference regions
JS-only appsSkipped unless useful content is present in HTML or embedded data

Use --strict-js-required when an agent should treat JS-only pages as hard errors instead of normal skips.

Output Formats

OutputUse it for
MarkdownLocal readable source snapshots with YAML frontmatter
NDJSONStreamed records or chunked records for agents and RAG
SQLiteLocal retrieval with an FTS5 index
OKFPortable Open Knowledge Format bundles with indexes and manifests
Archive / mirrorCached offline source snapshots

Every file-backed run writes corpus.manifest.json with stable document IDs, chunk IDs, hashes, output paths, and chunk counts. See Corpus Manifest.

Profiles

docpull https://site.com --profile rag        # Default. Dedup + metadata.
docpull https://site.com --profile llm        # NDJSON chunks for agents/RAG.
docpull https://site.com --profile okf        # Portable Open Knowledge Format bundle.
docpull https://site.com --profile mirror     # Cached archive.
docpull https://site.com --profile quick      # Small sampling crawl.
docpull https://site.com --profile sec-filing # EDGAR-friendly evidence chunks.

Run docpull --help for the full option list.

When Not to Use docpull

docpull intentionally does not use a browser. It is not the right tool for:

  • JS-only pages that require client-side rendering.
  • Authenticated dashboards or private apps.
  • Pages behind CAPTCHA or bot challenges.
  • Workflows that require clicking, scrolling, or browser state.

For those cases, use browser automation, such as Playwright, then pass the rendered HTML or exported content into your pipeline.

How It Compares

Tool typeBest forTradeoff
wget / site mirroringDownloading raw filesNot agent/RAG-oriented
Browser automationJS-heavy pages and interactionsSlower, heavier, more stateful
Hosted extraction APIsManaged extraction at scaleExternal dependency and cost
docpullLocal public-doc extraction and context packsNo JavaScript rendering

Python SDK

from docpull import fetch_one

ctx = fetch_one("https://docs.python.org/3/library/asyncio.html")
print(ctx.title)
print(ctx.markdown[:500])
import asyncio
from docpull import Fetcher, DocpullConfig, EventType, ProfileName

async def main():
    cfg = DocpullConfig(url="https://docs.example.com", profile=ProfileName.LLM)
    async with Fetcher(cfg) as fetcher:
        async for event in fetcher.run():
            if event.type == EventType.FETCH_PROGRESS:
                print(f"{event.current}/{event.total}: {event.url}")

asyncio.run(main())

MCP Server

docpull can run as a stdio MCP server for agent clients:

pip install 'docpull[mcp]'
docpull mcp

Claude Code:

claude mcp add --transport stdio docpull -- docpull mcp

Cursor and Claude Desktop use the same mcpServers shape:

{
  "mcpServers": {
    "docpull": {
      "type": "stdio",
      "command": "docpull",
      "args": ["mcp"]
    }
  }
}

The supported MCP path is the Python stdio server started by docpull mcp. The repository's mcp/ directory is an internal TypeScript/Bun lab and is not part of the package release contract.

Advanced Workflows

  • docpull[parallel] can discover, extract, enrich, score, diff, and archive live web sources with your own Parallel API key. See Parallel Integration.
  • Optional provider workflows can use Parallel, Tavily, and Exa when configured. See CLI Recipes.
  • SEC filing evidence packs use rule profiles such as vendor-dependency-rules.yml.

Security Defaults

  • HTTPS-only fetching with robots.txt compliance.
  • SSRF protections, private network blocking, DNS rebinding protection, and connect-time address pinning.
  • XXE protection for sitemaps.
  • Path traversal and CRLF header injection guards.
  • Auth headers stripped on cross-origin redirects.

When running with --proxy, DNS pinning is delegated to the proxy. Pass --require-pinned-dns to refuse that configuration.

Troubleshooting

docpull --doctor
docpull URL --verbose
docpull URL --dry-run
docpull URL --preview-urls

Documentation

Links

License

MIT

Ver o original no GitHub

Bom para

Relacionados