Web Search MCP
sydasif/web-search-mcp
13 ferramentas de busca e pesquisa web em tempo real para seus LLMs
Ver no GitHubServidor MCP que fornece acesso em tempo real a múltiplas fontes web para LLMs, incluindo buscas, leitura de artigos, Reddit, Hacker News, Wikipedia, GitHub, X/Twitter e mercados de previsão. Funciona sem chaves de API para a maioria das ferramentas e é pronto para produção.
Instalação
npx convez-ai sydasif/web-search-mcp1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Pesquisa em tempo real com dados atualizados para LLMs
- Monitoramento de discussões técnicas em comunidades (Reddit, Hacker News, GitHub)
- Análise de tendências e notícias recentes
- Sintetização de informações de múltiplas fontes web
Veredito

Ferrramenta sólida e production-ready que resolve o problema crítico de LLMs não terem acesso a dados atuais. Destaca-se pela cobertura ampla (13 tools), zero dependência de API keys para funcionalidade básica e integração simples com o protocolo MCP.
README
A tradução para o português deste README ainda está na fila.
Web Search MCP
13 tools that give your LLM real-time access to the web — from simple Google-style searches to deep AI-powered research. Search the web, read articles, check Reddit, browse Hacker News, look up Wikipedia, track GitHub issues, read full issue threads, follow X/Twitter discussions, explore prediction markets, and more.
No API keys required for most tools. Works out of the box.
What This Does
When you ask your LLM "what's the latest on X?", it can only answer from its training data. That data is months or years old. Web Search MCP fixes this by connecting your LLM to live web sources through the Model Context Protocol.
Here's what you get:
- Search the web — text and news search via DuckDuckGo
- Read any article — extract clean text from URLs, stripping ads and clutter
- Search Reddit — find real discussions and community sentiment
- Search Hacker News — developer opinions and tech discourse
- Search Wikipedia — factual summaries, background research, citations
- Search GitHub — find issues, PRs, and bug reports across repos
- Read GitHub issues — fetch full issue/PR threads with all comments, sorted by reactions
- Search X/Twitter — real-time posts and breaking news
- Search prediction markets — odds and crowd-sourced probability estimates from Polymarket
- AI-powered research — let Groq's models search, browse, and synthesize for you
Most tools are free and require no API keys. A few need setup (detailed below).
Quick Start (2 minutes)
Step 1: Install
uv tool install git+https://github.com/sydasif/web-search-mcp.git
Step 2: Add to Your MCP Client
Paste this into your MCP client config (Claude Desktop, VS Code, Cursor, etc.):
{
"mcpServers": {
"web-search": {
"command": "web-search-mcp"
}
}
}
That's it. You now have web search in your LLM.
Step 3: Optional — Enable More Tools
Want Reddit, Hacker News, GitHub, Polymarket, or X/Twitter search? They work out of the box — no extra config needed.
Want AI-powered research via Groq? Add an API key:
{
"mcpServers": {
"web-search": {
"command": "web-search-mcp",
"env": {
"SEARCH_MCP_GROQ_API_KEY": "gsk_your_key_here"
}
}
}
}
Get a free Groq API key here — Groq's free tier is generous.
What Can I Do With It?
Here are some real-world examples to get you started.
Search the Web
Ask your LLM: "Search for the latest Python 3.13 features"
Behind the scenes:
web_search("latest Python 3.13 features", search_type="text")
Read an Article
Ask your LLM: "Read this article and summarize it: https://docs.python.org/3/whatsnew/3.13.html"
Behind the scenes:
fetch_page(url="https://docs.python.org/3/whatsnew/3.13.html")
Search Reddit for Real Opinions
Ask your LLM: "What do people on Reddit think about Rust vs Go for web backends?"
Behind the scenes:
reddit_search(query="Rust vs Go web backend", subreddits=["rust", "golang"])
Search Hacker News for Tech Discourse
Ask your LLM: "Find Hacker News discussions about LLM fine-tuning in 2024"
Behind the scenes:
hackernews_search(query="LLM fine-tuning 2024")
Search Wikipedia
Ask your LLM: "What is the history of the Python programming language?"
Behind the scenes:
wikipedia_search(query="Python programming language")
This fetches the top matching article's full plain text (with section markers) plus a list of related articles. No API key needed.
Search GitHub Issues
Read a Full GitHub Issue Thread
Ask your LLM: "Read this issue and summarize the discussion: https://github.com/astral-sh/uv/issues/2000"
Behind the scenes:
get_github_issue(url="https://github.com/astral-sh/uv/issues/2000")
This fetches the entire thread — title, body, state, reactions, and all comments sorted by popularity — as structured Markdown. Works with both issues and pull requests. Requires gh CLI installed and authenticated, or GITHUB_TOKEN environment variable.
Search X/Twitter for Breaking News
Ask your LLM: "What's trending about AI on X right now?"
Behind the scenes:
x_search(query="AI", depth="default")
Search Prediction Markets
Ask your LLM: "What are the current odds on the next US presidential election?"
Behind the scenes:
polymarket_search(topic="US presidential election")
AI-Powered Deep Research
Ask your LLM: "Research the state of quantum computing in 2025 and validate the findings"
Behind the scenes:
groq_research(query="state of quantum computing 2025")
Choosing the Right Tool
Not sure which tool to use? Here's a quick guide:
| What You Want | Use This Tool | Why |
|---|---|---|
| Quick web search | web_search | Fast, free, no API key |
| Read a specific URL | fetch_page | Extracts clean text, strips ads |
| Search a specific site | search_docs | Adds site:domain.com automatically |
| Reddit discussions | reddit_search | Real community opinions |
| Tech opinions | hackernews_search | Developer-focused discussions |
| Factual summaries | wikipedia_search | Encyclopedia articles with full text |
| Bug reports / feature requests | github_search | Issues and PRs across repos |
| Full issue/PR conversation | get_github_issue | Complete thread with all comments |
| Real-time social media | x_search | Live posts and breaking news |
| Prediction market odds | polymarket_search | Crowd-sourced probabilities |
| Validate findings | groq_research | AI cross-checks and expands |
| Deep page analysis | groq_analyze_page | AI reads AND interprets |
How the Tools Connect
The best results come from combining tools. Here's the workflow that works best:
Step 1: Discover (free, fast) Use DuckDuckGo tools to find relevant sources quickly.
web_search("Python async performance benchmarks 2025")
Step 2: Read (free, fast) Pull the clean text from the most promising URLs.
fetch_page(url="https://example.com/benchmark-results")
Step 3: Validate (requires Groq API key) Let AI cross-check and expand on what you found.
groq_research("Validate these Python async benchmark findings and find any contradictions")
This discovery → reading → validation pattern gives you the most reliable results.
Environment Variables
| Variable | What It Does | Default | Required? |
|---|---|---|---|
SEARCH_MCP_GROQ_API_KEY | API key for Groq tools (browse, research, analyze_page) | — | Only for Groq tools |
SEARCH_MCP_RATE_LIMIT_SEARCH | Max DDG search requests per minute | 30 | No |
SEARCH_MCP_RATE_LIMIT_FETCH | Max page fetch requests per minute | 20 | No |
AUTH_TOKEN | X/Twitter session cookie | — | Only for x_search |
CT0 | X/Twitter CSRF token cookie | — | Only for x_search |
GITHUB_ISSUE_MAX_CHARS | Max characters for get_github_issue output | 30000 | No |
WIKIPEDIA_MAX_CHARS | Max characters for wikipedia_search output | 30000 | No |
Getting X/Twitter Cookies
The x_search tool needs two cookies from a logged-in X session. Here's how to get them:
- Log into x.com in your browser
- Open DevTools (press F12)
- Go to Application → Cookies →
https://x.com - Find and copy the values for
auth_tokenandct0 - Add them to your MCP config as
AUTH_TOKENandCT0
These cookies last about 24 hours. If X search stops working, refresh them.
Fetch Backend Options
The fetch_page tool can use different HTTP backends. Some websites block automated requests, so having options helps.
| Backend | How It Works | When to Use |
|---|---|---|
auto (default) | Tries httpx first, falls back to curl if blocked | Best for most situations |
httpx | Lightweight Python HTTP client | Fast, but some sites block it |
curl | Pretends to be Chrome's browser (TLS impersonation) | Bypasses Cloudflare and bot detection |
The auto backend handles this for you — it tries the fast option first and automatically switches if needed.
Tool Reference
DuckDuckGo Tools (free, no API key)
| Tool | What It Does | Key Parameters |
|---|---|---|
web_search | Search the web | query, search_type ("text" or "news"), max_results, time_range ("d", "w", "m", "y"), region, page |
fetch_page | Read a URL | url, output_format ("txt", "markdown", "html", "json"), include_metadata, max_length, backend |
search_docs | Search a specific site | query, domain (e.g., "docs.python.org") |
Reddit Search (free, no API key)
| Tool | What It Does | Key Parameters |
|---|---|---|
reddit_search | Search Reddit | query, subreddits (list), depth ("quick", "default", "deep"), time_range |
Hacker News Search (free, no API key)
| Tool | What It Does | Key Parameters |
|---|---|---|
hackernews_search | Search Hacker News | query, max_results, depth ("quick", "default", "deep") |
Wikipedia Search (free, no API key)
| Tool | What It Does | Key Parameters |
|---|---|---|
wikipedia_search | Search and read articles | query, max_results (default 5, max 20) |
GitHub Search (free, needs gh CLI or GITHUB_TOKEN)
| Tool | What It Does | Key Parameters |
|---|---|---|
github_search | Search Issues and PRs | query, max_results, depth, token (optional) |
get_github_issue | Fetch full issue/PR thread | url (full GitHub issue or PR URL) |
Polymarket Search (free, no API key)
| Tool | What It Does | Key Parameters |
|---|---|---|
polymarket_search | Search prediction markets | topic, max_results, depth |
X/Twitter Search (requires cookies)
| Tool | What It Does | Key Parameters |
|---|---|---|
x_search | Search X/Twitter | query, from_date (YYYY-MM-DD), depth |
Groq Tools (requires API key)
| Tool | What It Does | Best For |
|---|---|---|
groq_browse | Interactive multi-page web search | When you need the AI to explore multiple pages |
groq_research | Deep research — auto-selects tools | Validating and expanding on search findings |
groq_analyze_page | Read AND interpret a URL | Extracting specific insights from an article |
Which Groq model should I use?
groq/compound-mini(default) — Faster, cheaper, handles most tasks well. Start here.groq/compound— More thorough, uses more tools internally. Use for complex research.openai/gpt-oss-20b— Forgroq_browseonly. Good balance of speed and quality.openai/gpt-oss-120b— Forgroq_browseonly. Best quality, but slower.
Troubleshooting
"No results found" for web search
DuckDuckGo sometimes returns empty results for very specific or unusual queries. Try:
- Broadening your search terms
- Removing special characters
- Using a simpler query
Cloudflare blocks when reading URLs
This happens when a website detects automated requests. The fetch_page tool handles this automatically:
- Default backend (
auto) trieshttpxfirst, then falls back tocurlwith Chrome impersonation - If you're still blocked, try setting
backend="curl"explicitly
X/Twitter search not working
X cookies expire after ~24 hours. To fix:
- Log into x.com again
- Get fresh
auth_tokenandct0cookies (see Getting X/Twitter Cookies above) - Update your MCP config and restart the server
Groq tools returning errors
- Check that
SEARCH_MCP_GROQ_API_KEYis set correctly - Make sure your Groq API key has credits (free tier available)
- If you see "Query too long", shorten your query to under 150 characters
Rate limiting (429 errors)
DuckDuckGo will block you if you search too fast. The built-in rate limiter helps, but:
- Reduce
SEARCH_MCP_RATE_LIMIT_SEARCHif you're hitting limits - Add small delays between rapid searches
- Use
max_resultsto get fewer results per query
"get_github_issue" returns "gh CLI not installed"
The get_github_issue tool requires the GitHub CLI (gh) to fetch issue and PR threads:
# Install on Ubuntu/Debian
sudo apt install gh
# Install on macOS
brew install gh
# Authenticate (one time)
gh auth login
Alternatively, set GITHUB_TOKEN in your environment — gh will pick it up automatically without needing to run gh auth login. The gh binary itself is still required.
Server won't start
Make sure you're using uv (not pip):
# Install
uv tool install git+https://github.com/sydasif/web-search-mcp.git
# Run directly
uv run web-search-mcp
Frequently Asked Questions
Do I need API keys?
For most tools, no. Web search, Reddit, Hacker News, Wikipedia, GitHub, and Polymarket all work without any API key. You only need a Groq API key for the AI-powered tools (browse, research, analyze_page) and X/Twitter cookies for X search. The get_github_issue tool needs the gh CLI installed (which uses your existing GitHub auth) or a GITHUB_TOKEN environment variable.
Is this free?
The DuckDuckGo, Reddit, Hacker News, Wikipedia, GitHub, and Polymarket tools are completely free. Groq has a generous free tier. X/Twitter uses your browser session.
How is this different from just searching in the browser?
Your LLM can't access the web on its own. This server gives it that ability through the Model Context Protocol. The LLM can search, read, analyze, and synthesize information in a single conversation turn.
Can I use this with any LLM client?
Yes, as long as it supports MCP — Claude Desktop, VS Code with Copilot, Cursor, Zed, and more.
What about rate limits?
The server includes built-in rate limiting to avoid getting blocked by DuckDuckGo. You can configure the limits via environment variables (defaults: 30 searches/min, 20 fetches/min).
Development
Setup
git clone https://github.com/sydasif/web-search-mcp.git
cd web-search-mcp
uv sync
Run Tests
# All tests
uv run pytest
# With coverage
uv run pytest --cov=web_search_mcp
# Single test file
uv run pytest tests/test_models.py
# Specific test
uv run pytest tests/test_models.py::test_search_request_valid_defaults
Linting
uv run ruff check .
uv run ruff check --fix # auto-fix issues
Type Checking
uv run mypy web_search_mcp/
Run the Server
uv run web-search-mcp
License
Acknowledgment: This project is built on publicly available APIs, open-source libraries, and community research. DuckDuckGo, Reddit RSS, Hacker News Algolia API, Wikipedia MediaWiki API, GitHub REST API, Polymarket Gamma API, and Groq's API are all used in accordance with their respective terms.