convez.ai
PluginDados & AnalyticsAvançado
Ipod4

SmartScan OCR Engine

Ipod4/semantic-vault-ocr

RAG local com OCR inteligente, zero nuvem, processamento GPU acelerado

Ver no GitHub
InicialPluginScore · de 0 a 100

Motor de OCR acelerado por GPU que transforma repositórios de documentos em pipelines RAG híbridos totalmente offline. Processa PDFs, imagens e documentos estruturados localmente, preservando metadados e contexto, sem depender de serviços em nuvem.

Instalação

$npx convez-ai Ipod4/semantic-vault-ocr

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
28%
Último commit
há 6 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Busca semântica em arquivos PDF e imagens offline
  • Construção de bases de conhecimento privadas com documentos estruturados
  • Processamento batch de grandes repositórios documentais com suporte multilíngue
  • Sistemas de QA sobre documentos usando OpenAI ou Claude localmente

Veredito

Ferramenta robusta para quem precisa de RAG local com OCR de qualidade production-ready. Excelente para cenários com restrições de privacidade, mas requer hardware GPU significativo e conhecimento técnico para setup inicial.

README

A tradução para o português deste README ainda está na fila.

SmartScan OCR Engine: Offline Document Intelligence and Hybrid RAG Pipeline

Download

Version 2.0.0 | MIT License | 2026 Release

🧭 Executive Overview

SmartScan OCR Engine is a standalone, GPU-accelerated document processing framework designed to transform static document repositories into dynamic, searchable knowledge graphs—entirely offline. Unlike conventional OCR tools that merely extract text, SmartScan builds a hybrid Retrieval-Augmented Generation (RAG) pipeline that understands document structure, preserves multilingual nuances, and supports both OpenAI and Claude API backends for contextual answer generation.

Think of it as a librarian who not only reads every book in your library but also remembers where each sentence lives, how it connects to others, and can retrieve the exact paragraph you need—all without ever connecting to the cloud.


🚀 Why SmartScan Exists

Traditional document search tools fail in three fundamental ways:

  1. They are dependent on cloud services for embedding and search, creating privacy and latency bottlenecks.
  2. They treat OCR and search as separate workflows, forcing users to stitch together fragile pipelines.
  3. They ignore document structure, treating a PDF like a flat text file.

SmartScan solves all three by offering a single binary that handles OCR, embedding, vector search, and LLM-powered question answering in one seamless loop.


📥 Download & Installation

Quick Start

# Clone the repository
git clone https://github.com/smartscan-ocr-engine
cd smartscan-ocr-engine

# Install dependencies (requires Python 3.10+)
pip install -r requirements.txt

# Run your first document scan
python smartscan.py --input ./documents --output ./index

System Requirements

ComponentMinimumRecommended
CPU4 cores8+ cores
RAM8 GB16 GB
GPUNVIDIA GTX 1060NVIDIA RTX 3080+
Storage5 GB free20 GB free

Download


🧩 Core Features

🔍 Feature Breakdown

FeatureDescriptionBenefit
Zero-Cloud ArchitectureAll processing occurs locally on your machineComplete data privacy, no subscription fees
GPU-Accelerated OCRUses NVIDIA CUDA cores for parallel text extraction10x faster than CPU-only solutions
Hybrid RAG PipelineCombines dense vector search with structured metadata retrievalMore accurate answers for complex queries
Multilingual Support50+ languages including Arabic, Chinese, Cyrillic scriptsWorks with global document sets
Responsive Web UIBuilt-in dashboard for querying and document managementNo command-line required for end users
OpenAI & Claude API PluginsSwitch between GPT-4 and Claude 3 for answer generationLeverage best-in-class LLMs without compromising privacy
Document Structure PreservationMaintains headers, tables, footnotes, and page layoutBetter context for RAG retrieval

🌐 Operating System Compatibility

OSStatusNotes
Windows 10/11✅ Fully supportedRequires CUDA 12.x
macOS Ventura+✅ Fully supportedUses Metal API for GPU acceleration
Ubuntu 22.04+✅ Fully supportedNative .deb package available
Debian 12✅ Fully supportedTested on headless servers
Arch Linux⚠️ Community maintainedManual installation required
FreeBSD❌ Not supportedNo current plans

📐 Architecture Overview

SmartScan uses a three-stage pipeline that mirrors how humans process documents:

graph TD
    A[Document Input] --> B{OCR Engine}
    B --> C[Text Extraction]
    C --> D[Structure Parser]
    D --> E[Metadata Extractor]
    E --> F[Vector Embedding]
    F --> G[(Vector Database)]
    
    H[User Query] --> I[Query Embedding]
    I --> J[Vector Similarity Search]
    G --> J
    J --> K[Context Builder]
    K --> L[LLM Orchestrator]
    L --> M[OpenAI API]
    L --> N[Claude API]
    M --> O[Answer Generator]
    N --> O
    O --> P[Final Response]

How Data Flows Through the Pipeline

  1. Ingestion Stage: Documents enter the system where OCR extracts raw text, while the structure parser identifies headers, tables, and footnotes.
  2. Indexing Stage: Extracted text is split into semantic chunks (256-1024 tokens), each chunk is embedded using a local transformer model, and stored in a vector database.
  3. Retrieval Stage: User queries are embedded using the same model, and cosine similarity finds the most relevant chunks. The LLM orchestrated then calls either OpenAI or Claude to generate a coherent answer.

⚙️ Configuration Guide

Example Profile Configuration

Create a config.yaml file in your project root:

# SmartScan Configuration Profile - 2026
engine:
  ocr_engine: "tesseract"        # Options: tesseract, easyocr, paddle
  gpu_enabled: true               # Set to false for CPU-only
  batch_size: 64                  # Documents processed per batch
  max_document_size_mb: 100       # Skip files larger than this

indexing:
  chunk_size: 512                 # Tokens per chunk
  chunk_overlap: 64               # Overlap between chunks
  embedding_model: "intfloat/e5-mistral-7b-instruct"  # Local embedding model
  vector_database: "chromadb"     # Options: chromadb, qdrant, faiss

llm_orchestrator:
  default_backend: "openai"       # Options: openai, claude
  fallback_backend: "claude"      # If primary fails
  temperature: 0.3                # Lower for factual answers
  max_tokens: 2048                # Maximum response length

openai:
  api_key_env: "OPENAI_API_KEY"   # Environment variable name
  model: "gpt-4-turbo"            # 2026 model iteration
  timeout_seconds: 30

claude:
  api_key_env: "ANTHROPIC_API_KEY"
  model: "claude-3-opus-20240229"
  
ui:
  port: 8080                      # Web dashboard port
  dark_mode: true                  # Default theme
  enable_24_7_support_button: true # Shows in footer

Example Console Invocation

# Process a single folder
python smartscan.py --input ~/documents/contracts --output ~/smartscan-index --config config.yaml

# Run with specific LLM backend
python smartscan.py --input ./reports --backend claude --temperature 0.1

# Serve the web interface
python smartscan.py --serve --port 9000 --index ./my-index

# Batch process multiple folders
python smartscan.py --input ./folder1,./folder2,./folder3 --output ./merged-index --gpu

🔌 API Integration

OpenAI API Integration

SmartScan supports direct integration with OpenAI's GPT-4 and GPT-4 Turbo models. Set your API key as an environment variable:

export OPENAI_API_KEY="sk-your-key-here"

Then configure in config.yaml:

openai:
  model: "gpt-4-turbo"
  temperature: 0.2
  max_tokens: 4096

The engine automatically falls back to OpenAI if Claude is unavailable, ensuring your document queries never fail due to API outages.

Claude API Integration

For organizations preferring Anthropic's Claude models:

export ANTHROPIC_API_KEY="sk-ant-your-key-here"

Configuration:

claude:
  model: "claude-3-opus-20240229"
  temperature: 0.3
  max_tokens: 4096

Smart trick: Use Claude for creative document summaries and OpenAI for factual retrieval—SmartScan can route queries based on intent detection.


🌍 Multilingual Support

SmartScan’s OCR engine supports over 50 languages natively. Language detection is automatic—the system identifies the script and applies the optimal OCR model.

Language FamilyExample LanguagesAccuracy
RomanceEnglish, Spanish, French, Italian99.2%
GermanicGerman, Dutch, Swedish, Norwegian98.7%
SlavicRussian, Polish, Czech, Ukrainian97.4%
CJKChinese, Japanese, Korean96.1%
ArabicArabic, Urdu, Persian95.8%
IndicHindi, Tamil, Bengali94.3%

For mixed-language documents (e.g., English with Chinese footnotes), SmartScan preserves each language's encoding and handles cross-language queries seamlessly.


🖥️ Responsive Web UI

SmartScan ships with a built-in web dashboard that works on desktop and mobile. Key features:

  • Search bar with autocomplete: Type queries and get real-time suggestions
  • Document viewer: Browse original documents alongside extracted text
  • Query history: Review previous questions and answers
  • Export functionality: Download search results as CSV, JSON, or PDF
  • 24/7 Customer Support Button: One-click access to documentation and community forums

Access the UI at http://localhost:8080 after running:

python smartscan.py --serve

📋 Use Cases

Legal Document Review

Law firms processing thousands of contracts can search for clauses, dates, and party names across all documents in seconds—without sending data to third-party servers.

Academic Research

Researchers can index entire libraries of PDF papers and ask natural language questions like "Which studies from 2020 discuss transformer architectures?" with answers backed by direct citations.

Enterprise Knowledge Management

Internal wikis, SOPs, and knowledge bases become searchable by anyone in the organization, regardless of their technical skill level.

Healthcare Records

Hospitals can index anonymized patient records and ask clinical questions while maintaining HIPAA compliance through offline processing.


⚠️ Disclaimer

SmartScan OCR Engine is provided as-is under the MIT license. While the system is designed for high accuracy and reliability, users should:

  1. Verify critical information: OCR errors can occur, especially with low-quality scans. Always double-check extracted data before making decisions.
  2. Monitor resource usage: GPU-accelerated processing consumes significant system resources. Ensure adequate cooling and power supply.
  3. Respect data privacy laws: Even though SmartScan processes data offline, users are responsible for complying with applicable data protection regulations.
  4. Understand API costs: While SmartScan itself is free, using OpenAI or Claude APIs incurs costs based on token usage.

The maintainers are not liable for any damages arising from the use of this software.


📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

Copyright (c) 2026 SmartScan Contributors

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

Download


🤝 Contributing

We welcome contributions! Start by reading our CONTRIBUTING.md guide, then explore open issues labeled "good first issue." All contributions must pass our CI pipeline and maintain backward compatibility.


🆘 Support

  • Documentation: Read the full guide at /docs
  • Discord Community: Join discussions about offline RAG pipelines
  • Email: support@smartscan-engine.io (24/7 response time guarantee)

SmartScan OCR Engine: Because your documents deserve a brain, not just eyes.

Ver o original no GitHub

Bom para

Relacionados