convez.ai
SkillAutomaçãoIntermediário
Junhaozhang-127

Document Chunk Converter

Junhaozhang-127/document-chunk-converter

Converte documentos >32MB entre MD, PDF e DOCX via chunking inteligente

Ver no GitHub
InicialPluginScore · de 0 a 100

Skill para Claude Code que processa arquivos grandes (>32MB) dividindo-os em chunks, convertendo cada pedaço entre formatos MD/PDF/DOCX e remontando o resultado final. Resolve limitações de leitura de arquivos grandes e conversão de formatos de forma transparente e automática.

Instalação

$npx convez-ai Junhaozhang-127/document-chunk-converter

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
28%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Converter PDFs grandes para DOCX mantendo formatação
  • Processar relatórios volumosos que excedem limite de 32MB
  • Automatizar pipeline de conversão de formato para documentos em lote

Veredito

Solução pragmática para contornar limitações técnicas reais do Claude Code com arquivos grandes. Arquitetura modular e bem documentada, mas requer configuração específica por SO (especialmente Windows) e depende de ferramentas externas (pandoc, python-docx, weasyprint).

README

A tradução para o português deste README ainda está na fila.

Document Chunk Converter

English | 中文

A Claude Code Skill that transparently converts large document files (>32MB) between MD, PDF, and DOCX formats by slicing them into manageable chunks, converting each chunk, and reassembling the result.

Problem

Claude Code cannot read files larger than ~32MB. When format conversion is needed (e.g., PDF to DOCX), tools like pandoc or python-docx also struggle with large files. This Skill solves both problems: it automatically chunks large files below the read limit, converts each piece, and reassembles the output — all without user intervention.

Architecture

  large-file.pdf (>32MB)
         │
         ▼
  ┌─────────────────────────────────────────────────┐
  │  1. chunker.py                                   │
  │     • Detects file type (MD/PDF/DOCX)            │
  │     • Splits at natural boundaries where possible│
  │     • Falls back to fixed-size chunking if needed│
  │     • Output: chunks/ + manifest.json            │
  └──────────────────────┬──────────────────────────┘
         │
         ▼
  ┌─────────────────────────────────────────────────┐
  │  2. converter.py                                 │
  │     • Reads manifest.json                        │
  │     • Converts each chunk to target format       │
  │     • Marks failed chunks in manifest            │
  │     • Output: converted/ + manifest.json          │
  └──────────────────────┬──────────────────────────┘
         │
         ▼
  ┌─────────────────────────────────────────────────┐
  │  3. assembler.py                                 │
  │     • Merges all successful chunks               │
  │     • Skips failed chunks with warnings          │
  │     • Auto-cleans intermediate files             │
  │     • Output: output.docx (or .md/.pdf)           │
  └─────────────────────────────────────────────────┘

Installation

pip install -r requirements.txt

Windows-specific dependencies

For MD→PDF and DOCX→PDF conversions on Windows, install one of:

StrategyPackageNotes
Word COM (recommended)pywin32Requires Microsoft Word installed
WeasyPrintweasyprint (included)Requires GTK3 runtime
pdfkit (fallback)pdfkitRequires wkhtmltopdf

Usage

Three-step workflow

# Step 1: Chunk the large file into ~4MB pieces
python scripts/chunker.py large-input.pdf -o /tmp/lfc_chunks --max-mb 4

# Step 2: Convert each chunk to the target format
python scripts/converter.py /tmp/lfc_chunks/manifest.json docx -o /tmp/lfc_converted

# Step 3: Assemble converted chunks into final output
python scripts/assembler.py /tmp/lfc_converted/manifest.json -o output.docx

One-liner

python scripts/chunker.py big.pdf -o /tmp/lfc_chunks && \
python scripts/converter.py /tmp/lfc_chunks/manifest.json docx -o /tmp/lfc_converted && \
python scripts/assembler.py /tmp/lfc_converted/manifest.json -o output.docx

CLI Reference

chunker.py

FlagDescriptionDefault
-o, --output-dirOutput directory for chunks (required)—
-m, --max-mbMax chunk size in MB4.0
--manifestManifest filenamemanifest.json

converter.py

FlagDescription
-o, --output-dirOutput directory for converted chunks (required)
target_formatOne of: md, pdf, docx

assembler.py

FlagDescription
-o, --outputOutput file path (required)
--keep-chunksPreserve intermediate chunk files (default: auto-cleanup)

Supported Conversions

SourceTargetMethod
MDDOCXmarkdown → HTML → python-docx
MDPDFWord COM / weasyprint / pdfkit (3-strategy fallback)
DOCXMDpython-docx text extraction (headings + tables preserved)
DOCXPDFWord COM, fallback: DOCX → MD → PDF
PDFMDpdfplumber text extraction
PDFDOCXPDF → MD → DOCX (two-step)

Note: Two-step conversions (PDF→DOCX, DOCX→PDF fallback) use MD as an intermediate format. Some formatting may be simplified compared to direct conversion.

Chunking Strategy

The chunker uses a hybrid strategy to split files intelligently:

Level 1: Natural boundaries (preferred)

  • MD files: Splits at heading boundaries (#, ##, ###, etc.), keeping sections together
  • PDF files: Groups pages, estimating average page size to stay under the max chunk size
  • DOCX files: Splits at body element boundaries (paragraphs and tables), preserving document structure

Level 2: Fixed-size fallback

When a single natural segment (e.g., one very long section) exceeds the max chunk size, that segment is further split into fixed-size pieces. This ensures no chunk exceeds the limit while preserving structure wherever possible.

Each chunk's boundary_type (natural or fixed) is recorded in the manifest. During assembly, natural boundaries get proper separators (page breaks in DOCX, --- in MD).

Manifest Format

The manifest.json file is the data contract between pipeline stages:

{
  "source": "large-report.pdf",
  "source_type": "pdf",
  "total_chunks": 14,
  "chunks_dir": "/tmp/lfc_chunks",
  "target_format": "docx",
  "chunks": [
    {
      "index": 0,
      "file": "chunk_0000.pdf",
      "size_bytes": 3987456,
      "boundary_type": "natural",
      "page_range": [1, 12],
      "converted_file": "/tmp/lfc_converted/chunk_0000.docx",
      "status": "ok"
    }
  ]
}

Fields added by each stage:

  • chunker: source, source_type, total_chunks, chunks_dir, chunks[]
  • converter: target_format, chunks[].converted_file, chunks[].status
  • assembler: reads manifest, produces final output

Output Reporting

After a successful conversion, you'll see:

Assembly complete: output.docx
  Chunks: 14 merged, 0 skipped (of 14)

If some chunks failed to convert:

Assembly complete with warnings: output.docx
  Chunks: 12 merged, 2 skipped (of 14)
  Warning: 2 chunks could not be converted. Content may be incomplete.

Dependencies

Core (required for all conversions)

PackageVersionPurpose
python-docx≥0.8.11DOCX read/write, MD→DOCX
PyPDF2≥3.0.0PDF chunking, PDF assembly
pdfplumber≥0.10.0PDF text extraction
markdown≥3.5.0MD→HTML rendering
beautifulsoup4≥4.12.0HTML parsing for MD→DOCX
weasyprint≥60.0MD→PDF (fallback strategy)

Optional

PackagePurpose
markdown-it-pyAlternative MD→HTML renderer
pdfkitAlternative MD→PDF (requires wkhtmltopdf)
pywin32Windows Word COM automation (fastest PDF output)

Fault Tolerance

  • Failed conversions are non-fatal: The assembler skips chunks marked "status": "failed" and merges only successful ones
  • Partial output: If 2 of 14 chunks fail, you still get 12 chunks' worth of content in the output file
  • Failed chunks preserved: Use --keep-chunks to inspect failed chunks for debugging
  • Automatic cleanup: Intermediate files are removed after successful assembly (disable with --keep-chunks)

Limitations

  • Not for search/analysis: This Skill is for format conversion only. To search or edit large files, use grep, ripgrep, or split manually
  • Format fidelity: Two-step conversions (PDF→MD→DOCX, DOCX→MD→PDF) simplify formatting. Images, complex tables, and layout may not be perfectly preserved
  • Text-only PDF extraction: pdfplumber extracts text; images, charts, and vector graphics in PDFs are lost
  • No binary formats: Images, audio, video, and other binary formats are not supported

File Structure

document-chunk-converter/
├── SKILL.md                # Claude Code Skill definition
├── README.md               # English documentation
├── README_zh.md            # Chinese documentation
├── requirements.txt        # Python dependencies
├── .gitignore              # Ignores __pycache__, .tmp, etc.
└── scripts/
    ├── chunker.py          # Step 1: Smart file slicing
    ├── converter.py        # Step 2: Per-chunk format conversion
    └── assembler.py        # Step 3: Merge + cleanup

Common Issues

"Unsupported file type"

The chunker only accepts .md, .pdf, and .docx files. Check the file extension.

Missing dependency errors

If you see ImportError for a specific conversion, install the relevant optional package. The error message will tell you which one.

Chunks failing to convert

A single chunk may fail if it contains corrupt data or unsupported formatting. The assembler will skip it — inspect the chunk with --keep-chunks if needed.

Verification

To verify the Skill is correctly installed after setup, run:

python -m py_compile scripts/chunker.py scripts/converter.py scripts/assembler.py

All three should report no errors.

Ver o original no GitHub

Bom para

Relacionados