convez.ai
MCP ServerBusca & PesquisaIntermediário
sydasif

Web Search MCP

sydasif/web-search-mcp

13 ferramentas de busca e pesquisa web em tempo real para seus LLMs

Ver no GitHub
InicialPluginScore · de 0 a 100

Servidor MCP que fornece acesso em tempo real a múltiplas fontes web para LLMs, incluindo buscas, leitura de artigos, Reddit, Hacker News, Wikipedia, GitHub, X/Twitter e mercados de previsão. Funciona sem chaves de API para a maioria das ferramentas e é pronto para produção.

Instalação

$npx convez-ai sydasif/web-search-mcp

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
25
Forks
6
Acompanhando
0
Saúde
28%
Último commit
há 1 semana
Versão
v0.8.0
Issues abertas
1
Contribuidores
1

Casos de uso

  • Pesquisa em tempo real com dados atualizados para LLMs
  • Monitoramento de discussões técnicas em comunidades (Reddit, Hacker News, GitHub)
  • Análise de tendências e notícias recentes
  • Sintetização de informações de múltiplas fontes web

Veredito

Ferrramenta sólida e production-ready que resolve o problema crítico de LLMs não terem acesso a dados atuais. Destaca-se pela cobertura ampla (13 tools), zero dependência de API keys para funcionalidade básica e integração simples com o protocolo MCP.

README

A tradução para o português deste README ainda está na fila.

Web Search MCP

Python 3.11+ Code Style: Ruff FastMCP

13 tools that give your LLM real-time access to the web — from simple Google-style searches to deep AI-powered research. Search the web, read articles, check Reddit, browse Hacker News, look up Wikipedia, track GitHub issues, read full issue threads, follow X/Twitter discussions, explore prediction markets, and more.

No API keys required for most tools. Works out of the box.


What This Does

When you ask your LLM "what's the latest on X?", it can only answer from its training data. That data is months or years old. Web Search MCP fixes this by connecting your LLM to live web sources through the Model Context Protocol.

Here's what you get:

  • Search the web — text and news search via DuckDuckGo
  • Read any article — extract clean text from URLs, stripping ads and clutter
  • Search Reddit — find real discussions and community sentiment
  • Search Hacker News — developer opinions and tech discourse
  • Search Wikipedia — factual summaries, background research, citations
  • Search GitHub — find issues, PRs, and bug reports across repos
  • Read GitHub issues — fetch full issue/PR threads with all comments, sorted by reactions
  • Search X/Twitter — real-time posts and breaking news
  • Search prediction markets — odds and crowd-sourced probability estimates from Polymarket
  • AI-powered research — let Groq's models search, browse, and synthesize for you

Most tools are free and require no API keys. A few need setup (detailed below).


Quick Start (2 minutes)

Step 1: Install

uv tool install git+https://github.com/sydasif/web-search-mcp.git

Step 2: Add to Your MCP Client

Paste this into your MCP client config (Claude Desktop, VS Code, Cursor, etc.):

{
  "mcpServers": {
    "web-search": {
      "command": "web-search-mcp"
    }
  }
}

That's it. You now have web search in your LLM.

Step 3: Optional — Enable More Tools

Want Reddit, Hacker News, GitHub, Polymarket, or X/Twitter search? They work out of the box — no extra config needed.

Want AI-powered research via Groq? Add an API key:

{
  "mcpServers": {
    "web-search": {
      "command": "web-search-mcp",
      "env": {
        "SEARCH_MCP_GROQ_API_KEY": "gsk_your_key_here"
      }
    }
  }
}

Get a free Groq API key here — Groq's free tier is generous.


What Can I Do With It?

Here are some real-world examples to get you started.

Search the Web

Ask your LLM: "Search for the latest Python 3.13 features"

Behind the scenes:
  web_search("latest Python 3.13 features", search_type="text")

Read an Article

Ask your LLM: "Read this article and summarize it: https://docs.python.org/3/whatsnew/3.13.html"

Behind the scenes:
  fetch_page(url="https://docs.python.org/3/whatsnew/3.13.html")

Search Reddit for Real Opinions

Ask your LLM: "What do people on Reddit think about Rust vs Go for web backends?"

Behind the scenes:
  reddit_search(query="Rust vs Go web backend", subreddits=["rust", "golang"])

Search Hacker News for Tech Discourse

Ask your LLM: "Find Hacker News discussions about LLM fine-tuning in 2024"

Behind the scenes:
  hackernews_search(query="LLM fine-tuning 2024")

Search Wikipedia

Ask your LLM: "What is the history of the Python programming language?"

Behind the scenes:
  wikipedia_search(query="Python programming language")

This fetches the top matching article's full plain text (with section markers) plus a list of related articles. No API key needed.

Search GitHub Issues

Read a Full GitHub Issue Thread

Ask your LLM: "Read this issue and summarize the discussion: https://github.com/astral-sh/uv/issues/2000"

Behind the scenes:
  get_github_issue(url="https://github.com/astral-sh/uv/issues/2000")

This fetches the entire thread — title, body, state, reactions, and all comments sorted by popularity — as structured Markdown. Works with both issues and pull requests. Requires gh CLI installed and authenticated, or GITHUB_TOKEN environment variable.

Search X/Twitter for Breaking News

Ask your LLM: "What's trending about AI on X right now?"

Behind the scenes:
  x_search(query="AI", depth="default")

Search Prediction Markets

Ask your LLM: "What are the current odds on the next US presidential election?"

Behind the scenes:
  polymarket_search(topic="US presidential election")

AI-Powered Deep Research

Ask your LLM: "Research the state of quantum computing in 2025 and validate the findings"

Behind the scenes:
  groq_research(query="state of quantum computing 2025")

Choosing the Right Tool

Not sure which tool to use? Here's a quick guide:

What You WantUse This ToolWhy
Quick web searchweb_searchFast, free, no API key
Read a specific URLfetch_pageExtracts clean text, strips ads
Search a specific sitesearch_docsAdds site:domain.com automatically
Reddit discussionsreddit_searchReal community opinions
Tech opinionshackernews_searchDeveloper-focused discussions
Factual summarieswikipedia_searchEncyclopedia articles with full text
Bug reports / feature requestsgithub_searchIssues and PRs across repos
Full issue/PR conversationget_github_issueComplete thread with all comments
Real-time social mediax_searchLive posts and breaking news
Prediction market oddspolymarket_searchCrowd-sourced probabilities
Validate findingsgroq_researchAI cross-checks and expands
Deep page analysisgroq_analyze_pageAI reads AND interprets

How the Tools Connect

The best results come from combining tools. Here's the workflow that works best:

Step 1: Discover (free, fast) Use DuckDuckGo tools to find relevant sources quickly.

web_search("Python async performance benchmarks 2025")

Step 2: Read (free, fast) Pull the clean text from the most promising URLs.

fetch_page(url="https://example.com/benchmark-results")

Step 3: Validate (requires Groq API key) Let AI cross-check and expand on what you found.

groq_research("Validate these Python async benchmark findings and find any contradictions")

This discovery → reading → validation pattern gives you the most reliable results.


Environment Variables

VariableWhat It DoesDefaultRequired?
SEARCH_MCP_GROQ_API_KEYAPI key for Groq tools (browse, research, analyze_page)—Only for Groq tools
SEARCH_MCP_RATE_LIMIT_SEARCHMax DDG search requests per minute30No
SEARCH_MCP_RATE_LIMIT_FETCHMax page fetch requests per minute20No
AUTH_TOKENX/Twitter session cookie—Only for x_search
CT0X/Twitter CSRF token cookie—Only for x_search
GITHUB_ISSUE_MAX_CHARSMax characters for get_github_issue output30000No
WIKIPEDIA_MAX_CHARSMax characters for wikipedia_search output30000No

Getting X/Twitter Cookies

The x_search tool needs two cookies from a logged-in X session. Here's how to get them:

  1. Log into x.com in your browser
  2. Open DevTools (press F12)
  3. Go to Application → Cookies → https://x.com
  4. Find and copy the values for auth_token and ct0
  5. Add them to your MCP config as AUTH_TOKEN and CT0

These cookies last about 24 hours. If X search stops working, refresh them.


Fetch Backend Options

The fetch_page tool can use different HTTP backends. Some websites block automated requests, so having options helps.

BackendHow It WorksWhen to Use
auto (default)Tries httpx first, falls back to curl if blockedBest for most situations
httpxLightweight Python HTTP clientFast, but some sites block it
curlPretends to be Chrome's browser (TLS impersonation)Bypasses Cloudflare and bot detection

The auto backend handles this for you — it tries the fast option first and automatically switches if needed.


Tool Reference

DuckDuckGo Tools (free, no API key)

ToolWhat It DoesKey Parameters
web_searchSearch the webquery, search_type ("text" or "news"), max_results, time_range ("d", "w", "m", "y"), region, page
fetch_pageRead a URLurl, output_format ("txt", "markdown", "html", "json"), include_metadata, max_length, backend
search_docsSearch a specific sitequery, domain (e.g., "docs.python.org")

Reddit Search (free, no API key)

ToolWhat It DoesKey Parameters
reddit_searchSearch Redditquery, subreddits (list), depth ("quick", "default", "deep"), time_range

Hacker News Search (free, no API key)

ToolWhat It DoesKey Parameters
hackernews_searchSearch Hacker Newsquery, max_results, depth ("quick", "default", "deep")

Wikipedia Search (free, no API key)

ToolWhat It DoesKey Parameters
wikipedia_searchSearch and read articlesquery, max_results (default 5, max 20)

GitHub Search (free, needs gh CLI or GITHUB_TOKEN)

ToolWhat It DoesKey Parameters
github_searchSearch Issues and PRsquery, max_results, depth, token (optional)
get_github_issueFetch full issue/PR threadurl (full GitHub issue or PR URL)

Polymarket Search (free, no API key)

ToolWhat It DoesKey Parameters
polymarket_searchSearch prediction marketstopic, max_results, depth

X/Twitter Search (requires cookies)

ToolWhat It DoesKey Parameters
x_searchSearch X/Twitterquery, from_date (YYYY-MM-DD), depth

Groq Tools (requires API key)

ToolWhat It DoesBest For
groq_browseInteractive multi-page web searchWhen you need the AI to explore multiple pages
groq_researchDeep research — auto-selects toolsValidating and expanding on search findings
groq_analyze_pageRead AND interpret a URLExtracting specific insights from an article

Which Groq model should I use?

  • groq/compound-mini (default) — Faster, cheaper, handles most tasks well. Start here.
  • groq/compound — More thorough, uses more tools internally. Use for complex research.
  • openai/gpt-oss-20b — For groq_browse only. Good balance of speed and quality.
  • openai/gpt-oss-120b — For groq_browse only. Best quality, but slower.

Troubleshooting

"No results found" for web search

DuckDuckGo sometimes returns empty results for very specific or unusual queries. Try:

  • Broadening your search terms
  • Removing special characters
  • Using a simpler query

Cloudflare blocks when reading URLs

This happens when a website detects automated requests. The fetch_page tool handles this automatically:

  • Default backend (auto) tries httpx first, then falls back to curl with Chrome impersonation
  • If you're still blocked, try setting backend="curl" explicitly

X/Twitter search not working

X cookies expire after ~24 hours. To fix:

  1. Log into x.com again
  2. Get fresh auth_token and ct0 cookies (see Getting X/Twitter Cookies above)
  3. Update your MCP config and restart the server

Groq tools returning errors

  • Check that SEARCH_MCP_GROQ_API_KEY is set correctly
  • Make sure your Groq API key has credits (free tier available)
  • If you see "Query too long", shorten your query to under 150 characters

Rate limiting (429 errors)

DuckDuckGo will block you if you search too fast. The built-in rate limiter helps, but:

  • Reduce SEARCH_MCP_RATE_LIMIT_SEARCH if you're hitting limits
  • Add small delays between rapid searches
  • Use max_results to get fewer results per query

"get_github_issue" returns "gh CLI not installed"

The get_github_issue tool requires the GitHub CLI (gh) to fetch issue and PR threads:

# Install on Ubuntu/Debian
sudo apt install gh

# Install on macOS
brew install gh

# Authenticate (one time)
gh auth login

Alternatively, set GITHUB_TOKEN in your environment — gh will pick it up automatically without needing to run gh auth login. The gh binary itself is still required.

Server won't start

Make sure you're using uv (not pip):

# Install
uv tool install git+https://github.com/sydasif/web-search-mcp.git

# Run directly
uv run web-search-mcp

Frequently Asked Questions

Do I need API keys?

For most tools, no. Web search, Reddit, Hacker News, Wikipedia, GitHub, and Polymarket all work without any API key. You only need a Groq API key for the AI-powered tools (browse, research, analyze_page) and X/Twitter cookies for X search. The get_github_issue tool needs the gh CLI installed (which uses your existing GitHub auth) or a GITHUB_TOKEN environment variable.

Is this free?

The DuckDuckGo, Reddit, Hacker News, Wikipedia, GitHub, and Polymarket tools are completely free. Groq has a generous free tier. X/Twitter uses your browser session.

How is this different from just searching in the browser?

Your LLM can't access the web on its own. This server gives it that ability through the Model Context Protocol. The LLM can search, read, analyze, and synthesize information in a single conversation turn.

Can I use this with any LLM client?

Yes, as long as it supports MCP — Claude Desktop, VS Code with Copilot, Cursor, Zed, and more.

What about rate limits?

The server includes built-in rate limiting to avoid getting blocked by DuckDuckGo. You can configure the limits via environment variables (defaults: 30 searches/min, 20 fetches/min).


Development

Setup

git clone https://github.com/sydasif/web-search-mcp.git
cd web-search-mcp
uv sync

Run Tests

# All tests
uv run pytest

# With coverage
uv run pytest --cov=web_search_mcp

# Single test file
uv run pytest tests/test_models.py

# Specific test
uv run pytest tests/test_models.py::test_search_request_valid_defaults

Linting

uv run ruff check .
uv run ruff check --fix  # auto-fix issues

Type Checking

uv run mypy web_search_mcp/

Run the Server

uv run web-search-mcp

License

MIT License


Acknowledgment: This project is built on publicly available APIs, open-source libraries, and community research. DuckDuckGo, Reddit RSS, Hacker News Algolia API, Wikipedia MediaWiki API, GitHub REST API, Polymarket Gamma API, and Groq's API are all used in accordance with their respective terms.

Ver o original no GitHub

Bom para

Relacionados