convez.ai
AgenteAutomaçãoAvançado
hmshb

Scraping Agent AI

hmshb/scraping-agent-ai

Agent IA para web scraping inteligente com detecção de anti-bot

Ver no GitHub
InicialPluginScore · de 0 a 100

Agent de web scraping alimentado por IA que automatiza extração de dados de websites com crawling inteligente, detecção de anti-bot e parsing estruturado. Construído com LangGraph, Firecrawl e Claude, oferece tratamento adaptativo de erros, processamento em lote e múltiplos formatos de saída.

Instalação

$npx convez-ai hmshb/scraping-agent-ai

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
35
Forks
9
Acompanhando
2
Saúde
42%
Último commit
há 1 ano
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Extração em massa de dados estruturados de múltiplos websites
  • Coleta de inteligência competitiva e preços
  • Indexação de conteúdo web com formatação automática

Veredito

Agent robusto e bem arquitetado para scraping em escala, com bom suporte a anti-bot e processamento em lote. Exige familiaridade com LangGraph e APIs externas, adequado para projetos de extração de dados complexos.

README

A tradução para o português deste README ainda está na fila.

🚀 Scraping Agent AI

Python Web Scraping Automation

An AI-powered web scraping agent that automates data extraction from websites with intelligent crawling, anti-bot detection, and structured data parsing. Built using LangGraph, LangSmith, Firecrawl, and Anthropic AI tools for seamless AI-driven web scraping and structured data processing.

🔥 Features

  • Graph-based AI Agent: Uses LangGraph for managing scraping workflows.
  • Intelligent Web Crawling: Powered by Firecrawl to extract structured data.
  • LLM-Powered Formatting: Uses Anthropic AI for content summarization.
  • Adaptive Error Handling: Retries failed requests dynamically.
  • Batch Processing: Efficiently processes multiple URLs with batched requests.
  • Flexible Output Formats: Supports JSON, Markdown, and more.

Demo

demo.png


🛠️ Setup Instructions

Follow these steps to set up and run the project on your local machine:

1. Clone the Repository

git clone https://github.com/hmshb/scraping-agent-ai
cd scraping-agent-ai

2. Create a Virtual Environment

python -m venv venv
source venv/bin/activate  # For Linux/Mac

.\venv\Scripts\activate # For Windows

3. Install LangGraph CLI

pip install -U "langgraph-cli[inmem]"

img.png

4. Install Other Dependencies

pip install -e .

img_1.png


5. Generate LangSmith API Key

  1. Visit LangSmith.
  2. Create an API key for accessing LangSmith logs.
  3. Copy the generated API key.

img_2.png


6. Generate Anthropic Claude API Key

  1. Visit Anthropic.
  2. Create an API key for accessing Claude.
  3. Copy the generated API key.

img_3.png


7. Configure the Environment Variables

  • Copy the .env.example file and rename it to .env:
    cp .env.example .env
    
  • Open the .env file and update the API keys and configuration values:
    LANGSMITH_PROJECT=scrapping-agent
    LANGSMITH_API_KEY=your_api_key_here
    ANTHROPIC_API_KEY=your_api_key_here
    FIRECRAWL_API_KEY=your_api_key_here
    URL_LIMIT=10
    BATCH_LIMIT=5
    

8. Run the project

langgraph dev

img_4.png


9. LangGraph of the AI Agent

img_5.png


📂 Project Structure

scraping-agent-ai/
├── .env                 # API key configuration file
├── agent/               # Main AI scraping agent module
│   ├── utils/           # Utility modules for various tasks
│   │   ├── constants.py # Constants for scraping tasks
│   │   ├── firecrawl.py # Firecrawl integration
│   │   ├── graph.py     # LangGraph-based workflow
│   │   ├── helpers.py   # Utility functions
│   │   ├── llm.py       # LLM-powered formatting
│   │   ├── nodes.py     # Graph-based nodes
│   │   ├── states.py    # Scraping state management
│   ├── agent.py         # AI-driven scraping workflow
├── langgraph.json       # LangGraph configuration file
├── pyproject.toml       # Python project metadata
├── README.md            # Documentation file
├── scraped_data.json    # This will have the final data
├── venv/                # Virtual environment

⭐ Acknowledgments

Special thanks to:

  • LangGraph for building graph-based AI workflows.
  • LangSmith for debugging and monitoring AI agents.
  • Firecrawl for powerful web crawling and data extraction.
  • Anthropic AI for AI-powered text summarization and formatting.

📜 License

This project is open-source and licensed under the MIT License.


📢 Get Involved!

If you find this repository helpful, please consider:

  • ⭐ Starring the Repository to show your support.
  • 📤 Forking the Repository to explore further and make your own customizations.
  • 💬 Sharing Your Feedback by opening issues or discussions.

📝 Notes

LangGraph, LangSmith, Claude and FireCrawl is currently in limited or preview release (depending on your region and timing), and integration details may change as the service evolves.

Always refer to official documentation for the most up-to-date guidance.

Let's build smart, scalable AI-powered web scrapers together! 🚀

Ver o original no GitHub

Bom para

Relacionados