convez.ai
MCP ServerDados & AnalyticsAvançado
kreuzberg-dev

Kreuzcrawl

kreuzberg-dev/kreuzcrawl

Web crawling ultra-rápido em Rust com bindings para 11 linguagens

Ver no GitHub
BomPluginScore · de 0 a 100

MCP server de engine web crawling de alta performance escrito em Rust com suporte a 11 linguagens de programação. Fornece bindings multiplataforma (Python, Node.js, Go, Java, C#, PHP, Ruby, Elixir, Dart, Swift, Zig) para integração flexível em diversos ecossistemas.

Instalação

$npx convez-ai kreuzberg-dev/kreuzcrawl

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
197
Forks
30
Acompanhando
1
Saúde
62%
Último commit
há 2 dias
Versão
v1.13.1
Issues abertas
0
Contribuidores
8

Casos de uso

  • Scraping de dados em larga escala com performance crítica
  • Integração de crawling em pipelines de dados e analytics
  • Processamento paralelo de múltiplos URLs em aplicações web

Veredito

Ferramenta robusta para equipes que precisam de crawling de alto desempenho com flexibilidade multilíngue. A arquitetura em Rust garante velocidade, mas a curva de aprendizado é acentuada para iniciantes.

README

A tradução para o português deste README ainda está na fila.

Kreuzcrawl

High-performance Rust web crawling engine for structured data extraction. Scrape, crawl, and map websites with native bindings for 14 languages — same engine, identical results across every runtime.

Key Features

  • Structured extraction — Text, metadata, links, images, assets, JSON-LD, Open Graph, hreflang, favicons, headings, and response headers
  • Markdown conversion — Clean Markdown output with citations, document structure, and fit-content mode
  • Concurrent crawling — Depth-first, breadth-first, or best-first traversal with configurable depth, page limits, and concurrency
  • 14 language bindings — Rust, Python, Node.js, TypeScript, Ruby, Go, Java, Kotlin (Android), C#, PHP, Elixir, Dart, Swift, Zig, and WebAssembly
  • Smart filtering — BM25 relevance scoring, URL include/exclude patterns, robots.txt compliance, and sitemap discovery
  • Browser rendering — Optional headless browser for JavaScript-heavy SPAs with WAF detection and bypass
  • Batch operations — Scrape or crawl hundreds of URLs concurrently with partial failure handling
  • Streaming — Real-time crawl events via async streams for progress tracking
  • Authentication — HTTP Basic, Bearer token, and custom header auth with persistent cookie jars
  • Rate limiting — Per-domain request throttling with configurable delays
  • Asset download — Download, deduplicate, and filter images, documents, and other linked assets
  • MCP server — Model Context Protocol integration for AI agents
  • REST API — HTTP server with OpenAPI spec

Documentation | API Reference

Installation

LanguagePackageInstall
Pythonkreuzcrawlpip install kreuzcrawl
Node.js@kreuzberg/kreuzcrawlnpm install @kreuzberg/kreuzcrawl
Rustkreuzcrawlcargo add kreuzcrawl
Gopkg.go.devgo get github.com/kreuzberg-dev/kreuzcrawl/packages/go
JavaMaven CentralSee README
C#NuGetdotnet add package Kreuzcrawl
Rubykreuzcrawlgem install kreuzcrawl
PHPkreuzberg-dev/kreuzcrawlcomposer require kreuzberg-dev/kreuzcrawl
Elixirkreuzcrawl{:kreuzcrawl, "~> 0.2"}
WASM@kreuzberg/kreuzcrawl-wasmnpm install @kreuzberg/kreuzcrawl-wasm
C FFIGitHub ReleasesC header + shared library
CLIcrates.iocargo install kreuzcrawl-cli
CLI (Homebrew)kreuzberg-dev/tapbrew install kreuzberg-dev/tap/kreuzcrawl

Quick Start

from kreuzcrawl import create_engine, scrape

engine = create_engine()
result = scrape(engine, "https://example.com")

print(result.metadata.title)
print(result.markdown.content)
print(len(result.links))
import { createEngine, scrape } from "@kreuzberg/kreuzcrawl";

const engine = createEngine();
const result = await scrape(engine, "https://example.com");

console.log(result.metadata.title);
console.log(result.markdown.content);
console.log(result.links.length);
let engine = kreuzcrawl::create_engine(None)?;
let result = kreuzcrawl::scrape(&engine, "https://example.com").await?;

println!("{}", result.metadata.title);
println!("{}", result.markdown.content);
println!("{}", result.links.len());
engine, _ := kcrawl.CreateEngine()
result, _ := kcrawl.Scrape(engine, "https://example.com")

fmt.Println(result.Metadata.Title)
fmt.Println(result.Markdown.Content)
fmt.Println(len(result.Links))
var engine = Kreuzcrawl.createEngine(null);
var result = Kreuzcrawl.scrape(engine, "https://example.com");

System.out.println(result.metadata().title());
System.out.println(result.markdown().content());
System.out.println(result.links().size());
var engine = KreuzcrawlLib.CreateEngine(null);
var result = await KreuzcrawlLib.Scrape(engine, "https://example.com");

Console.WriteLine(result.Metadata.Title);
Console.WriteLine(result.Markdown.Content);
Console.WriteLine(result.Links.Count);
engine = Kreuzcrawl.create_engine(nil)
result = Kreuzcrawl.scrape(engine, "https://example.com")

puts result.metadata.title
puts result.markdown.content
puts result.links.length
$engine = Kreuzcrawl::createEngine(null);
$result = Kreuzcrawl::scrape($engine, "https://example.com");

echo $result->metadata->title;
echo $result->markdown->content;
echo count($result->links);
{:ok, engine} = Kreuzcrawl.create_engine(nil)
{:ok, result} = Kreuzcrawl.scrape(engine, "https://example.com")

IO.puts(result.metadata.title)
IO.puts(result.markdown.content)
IO.puts(length(result.links))

Platform Support

LanguageLinux x86_64Linux aarch64macOS ARM64Windows x64
Python✅✅✅✅
Node.js✅✅✅✅
WASM✅✅✅✅
Ruby✅✅✅—
Elixir✅✅✅✅
Go✅✅✅✅
Java✅✅✅✅
C#✅✅✅✅
PHP✅✅✅✅
Rust✅✅✅✅
C (FFI)✅✅✅✅
CLI✅✅✅✅

Guides

  • Observability — Integrate with Jaeger, Prometheus, or your observability backend. Emit and analyze crawl spans, metrics, and traces with W3C TraceContext propagation.
  • Antibot Strategy & Stealth — Detect and bypass WAF systems. Customize antibot behavior per vendor, enable stealth surfaces, and integrate third-party bypass providers.

Architecture

Your Application (Python, Node.js, Ruby, Java, Go, C#, PHP, Elixir, ...)
    │
Language Bindings (PyO3, NAPI-RS, Magnus, ext-php-rs, Rustler, cgo, Panama, P/Invoke)
    │
Rust Core Engine (async, concurrent, SIMD-optimized)
    │
    ├── HTTP Client (reqwest + tower middleware stack)
    ├── HTML Parser (html5ever + lol_html)
    ├── Markdown Converter (html-to-markdown-rs)
    ├── Content Extraction (metadata, JSON-LD, Open Graph, readability)
    ├── Link Discovery (robots.txt, sitemaps, anchor analysis)
    └── Browser Rendering (optional headless Chrome/Firefox)

Contributing

Contributions are welcome! See our Contributing Guide.

Part of Kreuzberg.dev

  • Kreuzberg — document intelligence: text, tables, metadata from 90+ formats with optional OCR.
  • Kreuzberg Cloud — managed extraction API with SDKs, dashboards, and observability.
  • html-to-markdown — fast, lossless HTML→Markdown engine.
  • liter-llm — universal LLM API client with native bindings for 14 languages and 143 providers.
  • tree-sitter-language-pack — tree-sitter grammars and code-intelligence primitives.
  • alef — the polyglot binding generator that produces all per-language bindings.
  • Discord — community, roadmap, announcements.

License

Elastic License 2.0

Links

Ver o original no GitHub

Bom para

Relacionados