convez.ai
MCP ServerBusca & PesquisaAvançado
brunowinter8192

SearXNG CLI — Busca Web e Scraping para Claude

brunowinter8192/searxng-cli

Busca web, scraping e crawl de sites com 8 engines + IA autônoma

Ver no GitHub
InicialPluginScore · de 0 a 100

MCP server que integra busca multi-mecanismo (8 engines) e scraping JavaScript-aware via Crawl4AI para Claude. Inclui exploração de sites, agente de pesquisa autônomo e pipeline de crawl com RAG.

Instalação

$npx convez-ai brunowinter8192/searxng-cli

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
1
Acompanhando
0
Saúde
28%
Último commit
há 4 dias
Versão
—
Issues abertas
4
Contribuidores
2

Casos de uso

  • Pesquisa autônoma aprofundada na web com múltiplas queries
  • Scraping de conteúdo dinâmico (JavaScript) e indexação em RAG
  • Crawl estruturado de documentações e sites inteiros

Veredito

Ferramenta sólida para pesquisa web automatizada e scraping inteligente. Bem pensada em features (multi-engine, JS-aware, agente autônomo), mas requer Python 3.10+ e setup inicial com Playwright — recomendado para quem precisa de pesquisa profunda integrada ao Claude.

README

A tradução para o português deste README ainda está na fila.

SearXNG

Web search, URL scraping, and site crawling for Claude Code — powered by a pydoll-based stealth browser engine.

Features

  • Multi-Engine Web Search — 8 search engines (Google, DuckDuckGo, Mojeek, Lobsters, Google Scholar, CrossRef, OpenAlex, Stack Exchange) with overlap-ranked, slot-allocated 20-URL output and disk-cached pagination via search_more
  • JavaScript-Aware Scraping — full page rendering via Crawl4AI with stealth fallback, cookie consent removal, and garbage detection
  • Site Exploration — sitemap and BFS-based URL discovery for crawl planning
  • Autonomous Research Agent — Haiku-powered agent that searches broadly, scrapes aggressively, and routes domain-specific results to specialized plugins (arXiv→RAG, GitHub→GitHub, Reddit→Reddit)

Quick Start

/plugin marketplace add brunowinter8192/claude-plugins
/plugin install searxng
# Restart session — mcp-start.sh bootstraps the Python venv automatically

Prerequisites

  • Python 3.10+
  • Playwright Chromium (auto-installed by mcp-start.sh)

mcp-start.sh bootstraps the Python venv and sets up Playwright Chromium on first run. No Docker required.

Setup

1. Clone + venv

git clone https://github.com/brunowinter8192/SearXNG.git
cd SearXNG
python -m venv venv
./venv/bin/pip install -r requirements.txt
./venv/bin/playwright install chromium

Usage

MCP Tools

ToolWhat it doesWhen to use
search_webSearch the web with language and time-range filters, returns top 20 URLs ranked across 8 engines (overlap + slot-allocation)Any web search
search_moreGet the next batch of URLs from the cached search (sha256-keyed disk cache, 1h TTL)Pagination beyond the first 20
scrape_urlFetch a page as filtered markdown with JS renderingRead full content of a search result
scrape_url_rawFetch a page as raw markdown, saved to filePrepare content for RAG indexing
explore_siteDiscover all URLs on a site via sitemap/BFSPlan a crawl before committing to it
download_pdfDownload a PDF file to local diskWhen a result URL ends in .pdf

Skills & Commands

  • /searxng:crawl-site — Crawl an entire website to markdown files. 6-phase pipeline: explore → assess → crawl → cleanup → index. Optionally spawns a RAG indexing worker (requires the RAG plugin with /rag:web-md-index).

Agents

  • web-research — Autonomous web research specialist. Searches with 5+ query variations, paginates through results, scrapes 10-15+ URLs per batch, and routes plugin-domain results (arXiv, GitHub, Reddit) to their respective MCP plugins. Dispatched automatically when deep web research is needed.

Workflows

Search → Scrape

  1. search_web with broad query → top 20 URLs ranked across 8 engines (overlap-counted across general engines, plus reserved slots for academic and Q&A engines), with search_more for the next batch from the disk cache
  2. Pick relevant URLs → scrape_url for full page content as markdown
  3. Plugin-domain URLs (arXiv, GitHub, Reddit) get routed to specialized plugins instead

Crawl → RAG

  1. /searxng:crawl-site https://docs.example.com — explore site structure
  2. Filter URL patterns (exclude noise like login pages, language variants)
  3. Crawl selected URLs → markdown files with garbage detection
  4. Optional: LLM cleanup (web-md-cleanup agent) → chunk + index into RAG

Troubleshooting

Some sites block headless browsers. The scraper tries normal mode first, then stealth mode with an undetected adapter. If both fail, the site likely has aggressive bot detection — try scrape_url_raw as an alternative.

mcp-start.sh installs Chromium automatically. For manual installation:

./venv/bin/playwright install chromium

License

MIT

Ver o original no GitHub

Bom para

Relacionados