Scraping Agent AI
hmshb/scraping-agent-ai
Agent IA para web scraping inteligente com detecção de anti-bot
Ver no GitHubAgent de web scraping alimentado por IA que automatiza extração de dados de websites com crawling inteligente, detecção de anti-bot e parsing estruturado. Construído com LangGraph, Firecrawl e Claude, oferece tratamento adaptativo de erros, processamento em lote e múltiplos formatos de saída.
Instalação
npx convez-ai hmshb/scraping-agent-ai1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Extração em massa de dados estruturados de múltiplos websites
- Coleta de inteligência competitiva e preços
- Indexação de conteúdo web com formatação automática
Veredito

Agent robusto e bem arquitetado para scraping em escala, com bom suporte a anti-bot e processamento em lote. Exige familiaridade com LangGraph e APIs externas, adequado para projetos de extração de dados complexos.
README
A tradução para o português deste README ainda está na fila.
🚀 Scraping Agent AI
An AI-powered web scraping agent that automates data extraction from websites with intelligent crawling, anti-bot detection, and structured data parsing. Built using LangGraph, LangSmith, Firecrawl, and Anthropic AI tools for seamless AI-driven web scraping and structured data processing.
🔥 Features
- Graph-based AI Agent: Uses LangGraph for managing scraping workflows.
- Intelligent Web Crawling: Powered by Firecrawl to extract structured data.
- LLM-Powered Formatting: Uses Anthropic AI for content summarization.
- Adaptive Error Handling: Retries failed requests dynamically.
- Batch Processing: Efficiently processes multiple URLs with batched requests.
- Flexible Output Formats: Supports JSON, Markdown, and more.

🛠️ Setup Instructions
Follow these steps to set up and run the project on your local machine:
1. Clone the Repository
git clone https://github.com/hmshb/scraping-agent-ai
cd scraping-agent-ai
2. Create a Virtual Environment
python -m venv venv
source venv/bin/activate # For Linux/Mac
.\venv\Scripts\activate # For Windows
3. Install LangGraph CLI
pip install -U "langgraph-cli[inmem]"

4. Install Other Dependencies
pip install -e .

5. Generate LangSmith API Key
- Visit LangSmith.
- Create an API key for accessing LangSmith logs.
- Copy the generated API key.

6. Generate Anthropic Claude API Key
- Visit Anthropic.
- Create an API key for accessing Claude.
- Copy the generated API key.

7. Configure the Environment Variables
- Copy the
.env.examplefile and rename it to.env:cp .env.example .env - Open the
.envfile and update the API keys and configuration values:LANGSMITH_PROJECT=scrapping-agent LANGSMITH_API_KEY=your_api_key_here ANTHROPIC_API_KEY=your_api_key_here FIRECRAWL_API_KEY=your_api_key_here URL_LIMIT=10 BATCH_LIMIT=5
8. Run the project
langgraph dev

9. LangGraph of the AI Agent

📂 Project Structure
scraping-agent-ai/
├── .env # API key configuration file
├── agent/ # Main AI scraping agent module
│ ├── utils/ # Utility modules for various tasks
│ │ ├── constants.py # Constants for scraping tasks
│ │ ├── firecrawl.py # Firecrawl integration
│ │ ├── graph.py # LangGraph-based workflow
│ │ ├── helpers.py # Utility functions
│ │ ├── llm.py # LLM-powered formatting
│ │ ├── nodes.py # Graph-based nodes
│ │ ├── states.py # Scraping state management
│ ├── agent.py # AI-driven scraping workflow
├── langgraph.json # LangGraph configuration file
├── pyproject.toml # Python project metadata
├── README.md # Documentation file
├── scraped_data.json # This will have the final data
├── venv/ # Virtual environment
⭐ Acknowledgments
Special thanks to:
- LangGraph for building graph-based AI workflows.
- LangSmith for debugging and monitoring AI agents.
- Firecrawl for powerful web crawling and data extraction.
- Anthropic AI for AI-powered text summarization and formatting.
📜 License
This project is open-source and licensed under the MIT License.
📢 Get Involved!
If you find this repository helpful, please consider:
- ⭐ Starring the Repository to show your support.
- 📤 Forking the Repository to explore further and make your own customizations.
- 💬 Sharing Your Feedback by opening issues or discussions.
📝 Notes
LangGraph, LangSmith, Claude and FireCrawl is currently in limited or preview release (depending on your region and timing), and integration details may change as the service evolves.
Always refer to official documentation for the most up-to-date guidance.