convez.ai
MCP ServerDados & AnalyticsAvançado
vikast908

Scrapo

vikast908/Scrapo

Web scraping agent-first com replay determinístico e extração LLM resiliente

Ver no GitHub
BomPluginScore · de 0 a 100

Biblioteca Python de web scraping nativa para IA com roteamento inteligente em 5 camadas (HTTP → browser → stealth → agent), extração híbrida com LLM e auto-recuperação de seletores, replay determinístico de todo fetch com auditoria de drift, e servidor MCP integrado para agentes.

Instalação

$npx convez-ai vikast908/Scrapo

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
4
Forks
1
Acompanhando
0
Saúde
71%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Extração de dados de sites dinâmicos para pipelines de IA
  • Crawling de grande escala com deduplicação e filas
  • Auditoria e replay de scraping com versionamento de resultados
  • Integração de coleta web em workflows de agentes autônomos

Veredito

Scrapo é uma solução madura e bem-arquitetada para scraping em escala de produção com foco em observabilidade e drift detection. Recomendado para equipes que precisam de auditoria completa de extrações e integração com pipelines de IA; exige familiaridade com Python async e gestão de agentes.

README

A tradução para o português deste README ainda está na fila.

🕸️ Scrapo

The web-scraping library agents deserve.

Selector-cheap. LLM-resilient. Replay-safe. Self-hosted.

CI Python License: MIT Status Type-checked Lint Async

Playwright Pydantic Anthropic OpenAI Gemini MCP

Quickstart | Architecture | Features | Why Scrapo | CLI | MCP


What is Scrapo?

Scrapo is a Python library that fuses four worlds the rest of the market keeps separate:

AI-native ingestionmarkdown, schema JSON

Agentic browsingobserve / act / extract

Production crawlingqueues, dedup, scaling

Managed accessproxies, anti-bot, geo

Plus a feature nobody else ships: deterministic replay of every fetch, so extraction drift is auditable.

Not a developer? layman.md explains what Scrapo does, and what it cannot do, in plain English.


Why Scrapo

5-tier routerHybrid extractorModel pinning
Auto-escalates HTTP, browser, stealth, agent on real failure signals onlySelector-cheap by default; falls back to LLM and self-healsStrict mode refuses unpinned LLM extraction, so extraction cannot silently drift
ProvenanceDeterministic replaySafe by default
Every chunk carries URL, selector path, byte range, heading trailRe-extract from archived HTML 6 months later; diff fields between any two runsSSRF guard, opt-in robots gate, regex+Luhn PII, geo allow/deny, append-only audit log

Architecture

                 +---------------------------------------------+
                 |               scrapo.scrape(...)            |
                 +-------------------+-------------------------+
                                     |
        +----------------------------+----------------------------+
        |                            |                            |
   (1) Tier Router            (2) Extractor             (3) Document Shaper
   T0 HTTP (retries)          Selector cache (host key)  HTML to Markdown
   T1 HTTP+session            -> if validation fails     Heading-aware chunks
   T2 Browser                 LLM fallback (budgeted)    Per-chunk provenance
   T3 Browser+stealth         -> self-heal / evict       Cross-crawl dedup
   T4 Agent                   selector cache writeback
        |                            |                            |
        +---------+------------------+---------+-------------------+
                  |                            |
            (4) Replay store             (5) Policy gate
            SQLite (WAL) + gzip HTML      SSRF / robots / PII / geo / audit
                  |                            |
                  +-------------+--------------+
                                |
                        (6) Agent surface
                        MCP server + tool schemas
scrapo/
├── access/      # (1) 5-tier router + pooled browser + request interception + agent driver + action cache + proxy adapters & rotating pool
├── extract/     # (2) hybrid selector + LLM (scalar & list fields), model pinning, cost-aware budget
├── shape/       # (3) markdown + heading chunker + content-type dispatch (HTML / JSON / feed / PDF / text)
├── replay/      # (4) SQLite metadata + pluggable snapshot store (local or S3) + field-level diff
├── policy/      # (5) robots, PII (flag or redact), geo, append-only audit
├── crawl/       # persistent SQLite queue + async scheduler + sitemap discovery + rel=next pagination
├── agent/       # (6) MCP server + tool schemas
├── results.py   # typed ScrapeResult / CrawlResult / ExtractionView
├── watch.py     # watch(url) -> Watch.refresh() -> ChangeSet (change tracking)
├── security.py  # SSRF guard for fetch targets
├── _db.py       # tuned SQLite connections (WAL, busy timeout)
├── logging.py   # structlog setup for the CLI / MCP server
├── api.py       # public scrape / extract / crawl / crawl_stream
├── web.py       # local browser UI (scrapo serve)
└── cli.py       # Typer CLI

Quickstart

pip install scrapo

pip install "scrapo[browser,anthropic,mcp]"
playwright install chromium

1. Scrape one URL

import asyncio, scrapo

async def main():
    res = await scrapo.scrape("https://example.com/")   # res is a typed ScrapeResult
    print(res.markdown)
    print("run_id:", res.run_id)
    # res["markdown"] / res.get("status") still work too (back-compat with the 0.1 dict)

asyncio.run(main())

2. Typed extraction, including lists (LLM once, selectors forever)

import asyncio, scrapo
from pydantic import BaseModel

class Offer(BaseModel):
    name: str
    price: str

class Listing(BaseModel):
    page_title: str
    offers: list[Offer] = []        # array fields become repeated-element extraction

async def main():
    res = await scrapo.scrape("https://example.com/shop", schema=Listing)
    print(res.extraction.data)      # {'page_title': '...', 'offers': [{'name': ..., 'price': ...}, ...]}
    print(res.extraction.method)    # 'llm' on the first run, 'selector' after
    print(res.cost_usd)             # 0.0 once selectors are cached

asyncio.run(main())

First call uses the LLM and caches the selectors it learns (keyed by host + schema; for list[Model] fields it caches a container selector plus per-subfield selectors). Every subsequent call against that host + schema uses cached selectors and zero LLM tokens. When the layout drifts, validation fails, Scrapo falls back to the LLM, re-derives selectors, and self-heals; a cache entry that keeps failing is evicted automatically.

3. Recursive crawl

await scrapo.crawl(
    seeds=["https://docs.python.org/3/"],
    max_depth=2,
    same_host_only=True,
)

4. Replay and diff

scrapo list                    # recent runs
scrapo replay <run_id>         # re-extract from archived HTML, no network
scrapo diff <run_a> <run_b>    # field-level diff

Features

TierWhat it doesWhen
T0 HTTPhttpx plain GET, with bounded retry/backoff on 429/5xx and transport errorsstatic HTML, JSON endpoints
T1 HTTP_SESSIONED+ browser-like headers/cookiessoft anti-bot
T2 BROWSERPlaywright headlessJS-rendered pages, SPA shells
T3 BROWSER_STEALTH+ stealth + residential proxyhard anti-bot
T4 AGENTLLM-driven multi-step browser via a pluggable AgentDriver (a reference LLMAgentDriver ships in; SCRAPO_AGENT_DRIVER=llm), with action caching so a repeated goal replays without the LLMlogins, captchas, flows

Escalation triggers: Cloudflare/Akamai/PerimeterX/DataDome/Distil fingerprints, HTTP 403 / 429 / 503, empty body, missing required schema fields, and unrendered single-page-app shells (lots of script, almost no rendered text). Budget(max_tier=..., max_llm_calls=..., max_cost_usd=...) caps how far it goes. The browser tiers block images/fonts/media/css by default and capture JSON XHR/fetch responses onto the result. The Tier-4 driver records the action sequence it used to reach a goal on a host (agent_actions.sqlite) and replays it on later runs with zero LLM tokens, self-healing back to the model only when a recorded step no longer applies (SCRAPO_AGENT_ACTION_CACHE=0 to disable).

A URL is not always an HTML page, so scrape() dispatches on Content-Type (with a little body sniffing):

ContentWhat you getresult.kind
text/htmlthe normal selectolax + markdown + chunk pipelinehtml
application/json / ld+jsonpretty-printed JSON as markdown; parsed object on result.datajson
RSS / Atoma markdown list of entries; parsed items on result.datafeed
application/pdfextracted text (requires pip install "scrapo[pdf]")pdf
text/plainthe body verbatimtext
cache hit + validates    -> return (method=selector, llm_calls=0, cost_usd=0)
miss / fail / over budget -> LLM with schema -> validate -> verify + persist selectors -> return (method=llm)
repeated cache failures   -> evict the stale entry, re-derive next run

The LLM is asked to return both the JSON payload and CSS selectors per field. A scalar field gets a string selector; a list[Model] field gets {"__list__": "<repeating element>", "<subfield>": "<selector relative to it>", ...}, which Scrapo applies as tree.css(container) then per-subfield extraction inside each match. Returned selectors are verified against the live HTML before being cached, so a hallucinated selector never poisons the cache. The cache is keyed by host (not registered domain), so blog.example.com and shop.example.com never collide.

from scrapo.extract.pinning import PinnedModel

pin = PinnedModel.make(
    provider="anthropic",
    model_id="claude-opus-4-7",
    prompt_template="(your prompt template)",
)

await scrapo.scrape(url, schema=Product, pin=pin, strict_pin=True)

strict_pin=True makes the extractor refuse to run if the configured LLM does not match the pin. Silent model drift cannot happen in production.

Every chunk Scrapo emits carries:

{
    "url":           "https://example.com/page",
    "selector_path": "markdown://Features/Pricing",
    "byte_start":    8421,
    "byte_end":      9842,
    "heading_trail": ["Features", "Pricing"],
    "chunk_hash":    "ab12cd34...",
}

You can trace any LLM citation back to a specific section of a specific URL.

Every fetch persists raw HTML, headers, screenshots, and the typed extraction:

scrapo replay 9f3e1c...        # re-extract from archived HTML, no network
scrapo diff 9f3e1c... abc123...  # field-level diff, with notes when model/schema changed

Sample diff output:

diff 9f3e1c...  vs  abc123...
  HTML changed
  ! model changed: anthropic:claude-opus-4-7 -> anthropic:claude-sonnet-4-6 (extraction may drift)
  field changes:
    - price: '$42' -> '$45'
    - in_stock: True -> False

scrapo scrape <url> --diff-last prints that diff against the previous run of the same URL in one step.

Re-scraping a URL the HTTP tier fetched before sends a conditional GET (If-None-Match / If-Modified-Since). A 304 Not Modified is rebuilt from the archived snapshot: no body transfer, no LLM call (the selector cache makes re-extraction free), and no duplicate snapshot is written. scrape() / crawl() get this automatically; Config(conditional_requests=False) (or SCRAPO_CONDITIONAL_REQUESTS=0) turns it off.

watch() builds the change-tracking loop on top of that:

import scrapo

w = await scrapo.watch("https://example.com/pricing", schema=Pricing)
# ... later, or on a schedule of your choosing ...
change = await w.refresh()
if change.not_modified:
    print("unchanged (304)")
elif change.changed:
    print(change.summary())          # field-level diff vs. the previous run
    for d in change.field_changes:
        print(d)                     # e.g. price: '$42' -> '$45'

Watch is in-process: the run history (and the diff) live in the replay store. Persisting a list of watches with a built-in scheduler is a hosted-service concern and is intentionally left out.

async for page in scrapo.crawl_stream(["https://blog.example.com/"], schema=Post):
    save(page)                       # process pages as they complete, not all at the end

Breaking out of the loop early stops the crawl and tears the shared browser down. crawl() remains the buffered convenience (returns aggregate stats + an on_page callback).

  • SSRF guard. Every fetch target is checked before a request goes out; loopback, link-local (including 169.254.169.254), private RFC 1918 / ULA ranges, and well-known local hostnames are refused. IP literals are parsed with inet_aton-style semantics, so obfuscated encodings (decimal 2130706433, hex 0x7f000001, short-form 127.1, dotted-octal 0177.0.0.1) of internal addresses are caught too. Tier-4 agent goto actions chosen by the LLM go through the same guard. Set allow_private_hosts=True (or SCRAPO_ALLOW_PRIVATE_HOSTS=1) for internal scraping. Crawl link discovery applies the same filter and skips obvious binary URLs.
  • Bounded HTTP retries. Transient 429 / 5xx and transport errors are retried with exponential backoff and jitter before the router escalates to a heavier tier (SCRAPO_HTTP_RETRIES, default 2).
  • Rotating proxy pool with health checks. Hand Scrapo a list of proxy URLs (Config(proxy_urls=[...]) or SCRAPO_PROXY_URLS="http://a,http://b") and the router round-robins across them. Every fetch's outcome is fed back: an HTTP 4xx auth/rate-limit code or an anti-bot fingerprint parks that proxy for proxy_cooldown_seconds (it's an IP-level block); a transient 5xx / network error counts toward max_failures; a clean fetch resets the streak. ProxyPool implements the ProxyAdapter protocol, so it composes with the vendor adapters; pass upstream=<adapter> to fall back to a managed gateway when every static endpoint is parked. Credentials are stripped from proxy URLs before they're logged.
  • Concurrency-safe storage. All SQLite stores (replay, selector cache, crawl queue, agent action cache) open in WAL mode with a busy timeout, and the per-store init step is guarded by an asyncio.Lock so concurrent first callers can't race on schema setup. Crawl workers honoring per-host crawl-delay no longer block workers for other hosts.
  • Pluggable snapshot storage. Replay metadata stays in SQLite, but the page bodies go through a SnapshotStore: local files by default (atomic write-then-rename, so a crash mid-write can't leave a partial snapshot recorded as complete), or S3 with snapshot_backend="s3://bucket/prefix" (pip install "scrapo[s3]"). A corrupt archive is detected on read and falls back to a fresh fetch instead of raising.
  • Browser reuse and lighter pages. A TierRouter launches one headless Chromium lazily and reuses it across fetches (proxy applied per context); the browser tiers also block images/fonts/media/css and surface JSON XHR responses. TierRouter.aclose() tears it down; scrape() and crawl() handle that for you.
  • Cost accounting. LLM cost is computed per call, recorded on the run, and enforceable via Budget(max_llm_calls=..., max_cost_usd=...).
  • PII handling. Flag PII in the audit log (SCRAPO_PII_FILTER=1), or redact it from the stored snapshot, markdown, and chunks (SCRAPO_REDACT_SNAPSHOTS=1).
  • Local UI hardening. scrapo serve binds 127.0.0.1 by default, validates the Host header against an allowlist (anti DNS-rebinding), serializes scrapes, and warns loudly if you bind a public interface.
  • Optional robots.txt parser with per-origin caching (off by default; set SCRAPO_RESPECT_ROBOTS=1)
  • Regex PII classifier (email, phone, SSN, credit card with Luhn, IPv4, IBAN, passport), flag or redact
  • Geo policy with EU-only preset (GeoPolicy.eu_only()) plus custom allow/deny lists
  • Append-only JSONL audit log of every scrape, block, geo violation, PII detection

You are responsible for complying with each site's terms of use and applicable law; these are tools, not a guarantee.

from scrapo.access.adapters.brightdata import BrightDataAdapter
import scrapo

adapter = BrightDataAdapter()  # reads BRIGHTDATA_USERNAME / BRIGHTDATA_PASSWORD
await scrapo.scrape("https://hard-target.com/", proxy_adapter=adapter)

Built-in: brightdata, oxylabs, scrapfly, zyte. Implement the ProxyAdapter protocol for anything else:

from scrapo.access.adapters.base import ProxyConfig

class MyAdapter:
    name = "my-vendor"
    async def get_proxy(self, geo=None):
        return ProxyConfig(url="http://user:pass@my-proxy:8080", region=geo)

Got your own list of proxies instead of a managed gateway? Use the built-in rotating pool: it round-robins, tracks per-endpoint health, and parks one that starts getting blocked:

from scrapo.access import ProxyPool

pool = ProxyPool(["http://u:p@a:8080", "http://u:p@b:8080"])     # or ProxyPool.from_env() / Config(proxy_urls=[...])
await scrapo.scrape("https://hard-target.com/", proxy_adapter=pool)
pool.stats()  # per-endpoint successes / failures / cooldown

Pass upstream=BrightDataAdapter() to ProxyPool to fall back to a managed gateway when every endpoint in the pool is cooling down.

ProviderAdapterInstallDefault model
Anthropic Claudeanthropic (default)pip install "scrapo[anthropic]"claude-opus-4-7
OpenAIopenaipip install "scrapo[openai]"gpt-4o-mini
Google Geminigeminipip install "scrapo[gemini]"gemini-2.5-flash
Mock (offline)mock (tests)always availablen/a

Anthropic adapter uses prompt caching on the schema block, so repeated extractions against the same Pydantic schema are cheap.


CLI

scrapo scrape https://example.com/
scrapo scrape https://example.com/ --max-tier 3 --screenshot --out-md page.md
scrapo crawl https://docs.python.org/3/ --max-depth 2 --max-pages 100
scrapo list --limit 10
scrapo replay <run_id>
scrapo diff <run_a> <run_b>
scrapo audit                    # tail the append-only audit log
scrapo adapters                 # list registered proxy adapters
scrapo serve                    # local browser UI at http://127.0.0.1:8787
scrapo mcp                      # run the MCP server over stdio

Use as an MCP server

Scrapo ships an MCP server exposing five tools to any MCP-compatible client (Claude Code, Claude Desktop, Cursor, and others):

scrapo_scrape    scrapo_crawl    scrapo_replay    scrapo_diff    scrapo_list_runs
pip install "scrapo[mcp]"
scrapo mcp

Add to your client config:

{
  "mcpServers": {
    "scrapo": {
      "command": "scrapo",
      "args": ["mcp"]
    }
  }
}

The SSRF guard is on by default, which matters here: an MCP client driven by an LLM that just read a page cannot be talked into fetching your internal services.


Configuration

Every default is overridable via env var:

VariableDefaultNotes
SCRAPO_DATA_DIRplatform user-data dirSQLite + snapshots + audit log
SCRAPO_USER_AGENTscrapo/0.1UA for HTTP and robots
SCRAPO_TIMEOUT30request timeout (s)
SCRAPO_CONCURRENCY8crawl concurrency
SCRAPO_HTTP_RETRIES2retries on 429/5xx/transport errors
SCRAPO_CONDITIONAL_REQUESTS10 to disable conditional GET / 304-archive reuse on re-scrapes
SCRAPO_RESPECT_ROBOTS01 to enable the robots gate
SCRAPO_PII_FILTER01 to flag PII in the audit log
SCRAPO_REDACT_SNAPSHOTS01 to redact PII from stored snapshots/markdown/chunks
SCRAPO_ALLOW_PRIVATE_HOSTS01 to allow fetching private/loopback addresses
SCRAPO_SNAPSHOT_BACKENDlocallocal or s3://bucket/prefix
SCRAPO_BROWSER_BLOCK_RESOURCES10 to let the browser tier load images/fonts/media/css
SCRAPO_BROWSER_CAPTURE_XHR10 to skip capturing JSON XHR/fetch responses
SCRAPO_AGENT_DRIVERunsetllm to enable the built-in Tier-4 agent driver
SCRAPO_AGENT_ACTION_CACHE10 to disable recording/replaying Tier-4 agent action sequences
SCRAPO_PROXY_ADAPTERunsetdefault registered adapter name
SCRAPO_PROXY_URLSunsetcomma-separated proxy URLs for the rotating pool (used when no adapter is set)
SCRAPO_PROXY_COOLDOWN120seconds a parked proxy stays out of rotation
SCRAPO_LLM_ADAPTERanthropicdefault LLM provider
SCRAPO_LLM_MODELclaude-opus-4-7default model id
SCRAPO_GEOunsetdefault proxy region
SCRAPO_LOG_LEVELINFOlog level for the CLI / MCP server
SCRAPO_LOG_FORMATconsoleconsole or json
ANTHROPIC_API_KEYunsetfor the Claude adapter
OPENAI_API_KEYunsetfor the OpenAI adapter
GEMINI_API_KEYunsetfor the Gemini adapter

Tests

pip install -e ".[dev]"
pytest -q
ruff check .
mypy scrapo/

The suite is fully offline; no test hits the network or a paid LLM. It covers signals (including SPA-shell detection), SSRF, the HTTP retry path, conditional GET / 304-archive reuse, watch() change tracking and crawl_stream, shape, extract (cache eviction, budget, cost), replay (and the schema migration), policy, dedup, queue, router, proxy adapters and the rotating pool (rotation, cooldown, hard vs. soft failures, upstream fallback, the tier feedback loop), the agent driver and its action cache (record / replay / self-heal / eviction with fake page + scripted LLM), the local web UI, config, and end-to-end scrape with monkeypatched fetchers.


Contributing

Issues and PRs welcome. See CONTRIBUTING.md for the dev setup and house rules, and SECURITY.md for reporting vulnerabilities.


Project status

Alpha. The public API (scrape, extract, crawl, crawl_stream, watch) is stable, as are tier escalation, model pinning, replay schema, typed results, list extraction, content-type routing, the pluggable snapshot store, the rotating proxy pool, conditional requests, and the MCP tool surface. The reference Tier-4 agent driver (with action caching: record the steps to a goal, replay them token-free, self-heal back to the LLM when a step breaks) and the in-browser request interception are functional but lightly exercised (a real browser is needed to validate them end to end). The library roadmap is otherwise complete; the only thing intentionally left out is a hosted control plane (a scheduler that runs and persists a list of watches, sends alerts, and gives you a web console). That would be a separate deployable service rather than part of the library.

See CHANGELOG.md for release notes.


License

MIT (c) Scrapo contributors

⭐ Star the repo if Scrapo saved you a week of glue code.

Ver o original no GitHub

Bom para

Relacionados