convez.ai
MCP ServerBusca & PesquisaIntermediário
quocthang0507

Local AI Web

quocthang0507/local-ai-web

Busca web e leitura de páginas para LLMs locais no LM Studio

Ver no GitHub
BomPluginScore · de 0 a 100

MCP server que integra busca web local, leitura de páginas HTML estáticas e renderização de SPAs com Playwright. Permite que LLMs locais (LM Studio) façam pesquisas na web usando SearXNG privado, buscar código em repositórios públicos e consultar documentos legais vietnamitas oficiais, tudo sem deixar o computador.

Instalação

$npx convez-ai quocthang0507/local-ai-web

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
85%
Último commit
há 3 meses
Versão
—
Issues abertas
21
Contribuidores
2

Casos de uso

  • Pesquisa web privada para modelos de linguagem local
  • Busca de código real em GitHub/StackOverflow para reduzir alucinação
  • Pesquisa e extração de documentos legais vietnamitas oficiais
  • Leitura de páginas HTML renderizadas com JavaScript

Veredito

Ferramenta sólida e bem-arquitetada para expandir capacidades de pesquisa em modelos locais, com foco em privacidade e uso off-line. Inclui nichos específicos (legislação vietnamita, busca de código) que agregam valor real, mas requer setup cuidadoso com SearXNG e pode ter curva de aprendizado para quem não usa LM Studio.

README

A tradução para o português deste README ainda está na fila.

local-ai-web

Local web search and web reading tools for local LLMs running in LM Studio.

local-ai-web adds a small MCP server to LM Studio so your local model can:

  • Search the web through a local SearXNG instance.
  • Read normal static HTML pages.
  • Render JavaScript-heavy SPA pages with Playwright Chromium.
  • Extract visible text, rendered HTML, and cleaned Markdown.
  • Block localhost/private-network requests to reduce SSRF risk.

Web Code Search (Internet)

This project now supports real-time code search from the Internet via the search_code_web tool.

It allows local LLMs (Qwen, Llama, etc.) to:

  • find real-world code examples from GitHub / StackOverflow / docs
  • extract working code snippets
  • reduce hallucination significantly

👉 Works completely local + private (self-hosted SearXNG)

Vietnam Legal & Administrative Document Research

This project also includes specialized retrieval tools for Vietnamese law, administrative documents, and public procedures:

  • search_vietnam_legal: searches official Vietnamese sources first (vbpl.vn, vanban.chinhphu.vn, congbao.chinhphu.vn, *.gov.vn, quochoi.vn).
  • fetch_vietnam_legal_document: fetches a legal/admin document URL and extracts text, Markdown, and heuristic metadata such as document number, authority, issue date, effective-date signals, and citations.
  • vietnam_legal_qa_context: builds source-backed context for Q&A by searching official sources and fetching top documents.

These tools are designed for retrieval and source grounding. They do not replace a qualified Vietnamese lawyer or a competent authority, and models should cite URLs and state uncertainty when legal effect is not verified.

Translation

The project includes integrated text translation for both the API/web app and LM Studio MCP:

  • translate_text: translates text through free/no-key external providers.
  • REST endpoint: POST /api/translate.
  • Web mode: Dịch thuật.

Supported providers:

  • auto: tries configured providers in order.
  • google: uses an unofficial Google Translate endpoint.
  • mymemory: uses the public MyMemory translation API.
  • duckduckgo: experimental best-effort DuckDuckGo Instant Answer lookup. It often does not return translations and is mostly a fallback.
  • lingva: uses the public Lingva Translate API (privacy-focused Google Translate proxy).

Do not send secrets or sensitive personal data to external translation providers.

This project is for local research and personal productivity. It does not bypass login pages, paywalls, CAPTCHA, or website access controls.


Table of Contents


Architecture

local-ai-web has two local entry points:

  • A browser web app served by the Express API server at http://localhost:3000.
  • An MCP stdio server launched by LM Studio through mcp.json.

Both entry points share the same retrieval, extraction, ranking, cache, and safety modules.

flowchart LR
    Browser[Browser Web App] --> API[Express Web/API Server]
    LMStudio[LM Studio / Local LLM] --> MCP[MCP stdio Server]

    API --> Tools[Shared Tool Modules]
    MCP --> Tools

    Tools --> SearXNG[Local SearXNG Docker]
    Tools --> SafeFetch[Safe HTTP Fetch]
    Tools --> Renderer[Playwright Renderer]
    Tools --> Cache[In-memory Cache]

    SafeFetch --> Extractors[Text / Markdown / PDF / Tables]
    Renderer --> Extractors

    Extractors --> API
    Extractors --> MCP

The shared tool layer provides these specialized retrieval pipelines:

flowchart TD
    Q[User Query] --> Mode{Search Mode}

    Mode --> Web[Web Search]
    Web --> WebSearch[SearXNG Search]
    WebSearch --> WebResults[Compact Results]
    WebResults --> WebFetch[Fetch Detail on Click]

    Mode --> Code[Code Search]
    Code --> CodeRewrite[Query Rewriting]
    CodeRewrite --> CodeSearch[SearXNG Search]
    CodeSearch --> CodeFilter[Prioritize GitHub / StackOverflow / Docs]
    CodeFilter --> CodeExtract[Extract Code Blocks]
    CodeExtract --> CodeRank[Rank Snippets]

    Mode --> Legal[Vietnam Legal / Admin Search]
    Legal --> LegalRewrite[Legal Query Rewriting]
    LegalRewrite --> LegalSearch[Official-domain Search]
    LegalSearch --> LegalFetch[Fetch HTML / PDF Document]
    LegalFetch --> LegalMeta[Extract Legal Metadata and Excerpts]

    WebFetch --> UI[Optimized Web UI]
    CodeRank --> UI
    LegalMeta --> UI

SearXNG runs in Docker and stays bound to 127.0.0.1. The Node.js server exposes REST APIs, Swagger UI, and the browser app. The MCP server exposes the same capabilities to LM Studio. All outbound fetch/render paths go through SSRF protection, truncation, and untrusted-content handling before returning data to a model or the web UI.


Features

health_check

Checks whether the MCP server, SearXNG, and Playwright Chromium are ready.

search_web

Searches the web through local SearXNG and returns compact results. Supports time range filtering (day, week, month, year) for finding recent information:

{
  "title": "...",
  "url": "...",
  "snippet": "...",
  "source": "example.com"
}

fetch_url

Fetches normal static HTML or plain text pages as plain text. Automatically converts GitHub/Gist file URLs to raw content for clean code reading.

fetch_markdown

Fetches static HTML pages and returns cleaned, well-formatted Markdown. Automatically converts GitHub/Gist file URLs to raw content and wraps them in code blocks.

Use this for:

  • Reading blog articles or documentation.
  • Getting high-quality content without boilerplate (headers, footers, ads).
  • Preserving tables and formatting via GFM (GitHub Flavored Markdown).

list_github_repo

Lists files and directories in a GitHub repository or subdirectory. Useful for exploring repository structures before reading specific files.

fetch_rendered_source

Renders JavaScript-heavy pages in headless Chromium and returns visible text and/or rendered DOM HTML.

Use this for:

  • React apps.
  • Vue apps.
  • Angular apps.
  • Next.js/Nuxt pages.
  • SPA pages where normal HTTP fetch only returns an empty shell.

fetch_rendered_markdown

Renders a page with Chromium, extracts the main content, and converts it to Markdown.

This is usually the best tool for LLM summarization.

fetch_document

Fetches and extracts raw text from PDF files and other supported documents.

Use this for:

  • Research papers (PDFs).
  • Manuals and technical documentation.

extract_structured_data

Extracts high-fidelity data that often gets lost in standard text conversion. Automatically detects HTML tables and converts them to Markdown, and extracts JSON-LD metadata. Supports an optional render flag for SPAs.

Use this for:

  • Extracting tables of data (e.g., pricing, specifications).
  • Reading product or recipe metadata.

OpenAPI API Server

Includes a standalone Express server providing REST APIs and a Swagger UI for executing all web search and reading features independently of LM Studio.

Angular SPA Web App

Includes a modern Single Page Application (SPA) web frontend built using Angular and styled with Pico.css v2 for clean, responsive, semantic aesthetics. It is served by the same Express server.

To access the web app, run the container stack or start the local API server and open:

http://localhost:3000

The app features:

  • Search modes: Tabbed search interface for Web, Mã nguồn (Code), Văn bản pháp luật (Legal Documents), Thủ tục hành chính (Administrative Procedures), and PDF.
  • Integrated Translation: A dedicated translation page supporting multiple free translation APIs (including Google Translate, MyMemory, and Lingva Translate) with a persistent translation history log saved in your browser.
  • Optimized UI: Responsive dual-pane layout showing results and detailed previews cleanly side-by-side.

Swagger remains available at:

http://localhost:3000/docs

clear_cache

Clears in-memory cache.

cache_stats

Shows in-memory cache statistics (e.g. memory usage, cache size).

close_browser

Closes the background Playwright Chromium browser instance to free up memory.

translate_text

Translates text using free/no-key external providers. Provider auto tries the configured provider list; google and mymemory are the practical defaults, while duckduckgo is experimental.

search_code_web

  • Web code search
  • multi-query search via SearXNG
  • GitHub raw code extraction
  • StackOverflow snippet extraction
  • automatic ranking (best snippet first)

search_vietnam_legal

Searches official Vietnamese legal/admin sources using retrieval-friendly query rewrites. Use this before answering questions about Vietnamese law, administrative-document format, or public procedures.

fetch_vietnam_legal_document

Fetches a Vietnamese legal/admin URL and returns extracted text, Markdown, and heuristic metadata including document number, document type, issuing authority, date signals, status signals, and cited documents.

vietnam_legal_qa_context

Builds source-backed context for Vietnamese legal/admin Q&A by combining official-source search with top-document fetching and relevant excerpt selection.


Project Structure

Recommended structure:

local-ai-web/
├─ README.md
├─ SECURITY.md
├─ .env.example
├─ .gitignore
├─ docker-compose.yml      # Starts SearXNG + mcp-web-reader together
├─ searxng/
│  └─ config/
│     └─ settings.yml
├─ mcp-web-reader/
│  ├─ package.json
│  ├─ tsconfig.json
│  ├─ Dockerfile           # Multi-stage Docker image
│  ├─ src/
│  │  ├─ index.ts          # Server entry point
│  │  ├─ browser.ts        # Playwright browser management
│  │  ├─ tools.ts          # MCP tool definitions & handlers
│  │  ├─ searxng.ts        # SearXNG API client
│  │  ├─ code_web.ts       # Code search & extraction logic
│  │  ├─ legal_vn.ts       # Vietnam legal/admin search & extraction logic
│  │  ├─ translate.ts      # Translation provider integration
│  │  ├─ structured.ts     # Table & Metadata extraction
│  │  ├─ cache.ts          # In-memory caching
│  │  ├─ extract.ts        # HTML to Markdown/Text extraction
│  │  ├─ config.ts         # Environment configuration
│  │  ├─ helper.ts         # Utility functions
│  │  ├─ security.ts       # SSRF & domain filtering
│  │  ├─ log.ts            # Logging utilities
│  │  ├─ http.ts           # Standalone API Server
│  │  └─ swagger.ts        # OpenAPI Specification
│  ├─ public/              # Compiled static Angular SPA assets
│  ├─ frontend/            # Angular SPA source project folder
│  └─ scripts/
│     ├─ print-mcp-config.js
│     └─ verify.js
└─ examples/
   ├─ prompts.md
   ├─ mcp.windows.json
   ├─ mcp.macos.json
   └─ mcp.linux.json

How to Set Up

Please refer to SETUP.md for detailed instructions on:

  • Requirements
  • Installing Docker
  • Installing Node.js
  • Setting up SearXNG
  • Setting up the MCP Server
  • Configuring LM Studio

How to Use

Please refer to USAGE.md for detailed instructions on:

  • Searching code snippets
  • Recommended System Prompts
  • GitHub Raw Optimization
  • Example prompts for each feature

Development Workflow

Start SearXNG:

docker compose up -d

Stop SearXNG:

docker compose down

Rebuild MCP after backend changes:

cd mcp-web-reader
npm run build

Rebuild Angular frontend SPA:

cd mcp-web-reader/frontend
npm run build

Run Angular frontend in dev mode (runs server on port 4200, proxies requests to port 3000):

cd mcp-web-reader/frontend
npm start

Watch TypeScript changes:

npm run build:watch

build:watch rebuilds dist/index.js, but the running MCP process still needs to be restarted or reconnected in LM Studio.


Troubleshooting

TS7016: Could not find a declaration file for module 'jsdom'

npm install -D @types/jsdom
npm run build

SearXNG returns 403 Forbidden for JSON

Make sure searxng/config/settings.yml contains:

search:
  formats:
    - html
    - json

Restart SearXNG:

docker compose restart searxng

SearXNG logs many 403 / Too many request / CAPTCHA engine errors

Some upstream engines rate-limit or CAPTCHA data-center/shared IPs aggressively. You can limit MCP queries to a smaller set of engines:

SEARXNG_ENGINES=google,bing

Then restart MCP (or reconnect in LM Studio).

If SEARXNG_ENGINES is not set, MCP automatically uses bing,duckduckgo,wikipedia.

Port 8080 is already in use

Edit .env:

SEARXNG_PORT=8888
SEARXNG_URL=http://127.0.0.1:8888

Restart:

docker compose down
docker compose up -d

Regenerate LM Studio config:

cd mcp-web-reader
npm run print:mcp

LM Studio does not show tools

cd mcp-web-reader
npm run build
npm run print:mcp

Make sure mcp.json uses the absolute path to dist/index.js.

LM Studio connects but tools do not respond or the model unloads

Use the config generated by npm run print:mcp again. It now defaults to a smaller LM Studio tool set:

MCP_ENABLED_TOOLS=health_check,search_web,fetch_url,fetch_markdown,fetch_document,extract_structured_data,translate_text,close_browser
HEALTH_CHECK_BROWSER=0
REQUEST_TIMEOUT_MS=10000
SEARXNG_ENGINES=bing,duckduckgo,wikipedia

This avoids loading large browser-rendering tool schemas into LM Studio and prevents health_check from launching Chromium. To expose every MCP tool, set MCP_ENABLED_TOOLS to an empty string. To include Playwright in health_check, set HEALTH_CHECK_BROWSER=1.

Also verify Docker Desktop is running and SearXNG responds at:

http://127.0.0.1:8080/search?q=test&format=json

Playwright browser is missing

npx playwright install chromium

Linux:

npx playwright install --with-deps chromium

Static fetch returns an empty shell

Use fetch_rendered_source or fetch_rendered_markdown.

Rendered fetch misses lazy-loaded content

Try higher wait_ms and scroll_steps:

Use fetch_rendered_markdown with wait_ms=5000 and scroll_steps=8.

Security Notes

Do not add shell or filesystem tools to this MCP server unless you fully understand the risk.

Keep SearXNG bound to 127.0.0.1 unless you know what you are doing.

For high-trust workflows, set ALLOW_DOMAINS:

ALLOW_DOMAINS=lmstudio.ai,github.com,docs.searxng.org

Treat every web page as untrusted input.


Limitations

This project cannot reliably extract:

  • Login-only content.
  • CAPTCHA-protected content.
  • Heavily anti-bot-protected content.
  • Paywalled content.
  • Canvas/WebGL-only text.
  • Content requiring complex user interaction.
  • Some websites block scraping
  • Dynamic sites may require rendering
  • Not all pages contain clean code blocks
  • GitHub issues are ignored

It does not bypass website access controls.


Useful Links


License

MIT

Ver o original no GitHub

Bom para

Relacionados