convez.ai
PluginDados & AnalyticsAvançado
thepapermasters

Bedrock OpenSearch - Extrator de PDFs com IA

thepapermasters/bedrock-opensearch

Extrai, processa e indexa PDFs com Claude + OpenSearch

Ver no GitHub
InicialPluginScore · de 0 a 100

Plugin que extrai imagens de PDFs, processa informações estruturadas usando Claude via Bedrock, e indexa os resultados como embeddings no OpenSearch. Implementa uma arquitetura serverless com AWS Batch, SQS e Lambda para processamento em escala de documentos com análise por IA.

Instalação

$npx convez-ai thepapermasters/bedrock-opensearch

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
50%
Último commit
há 2 anos
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Busca semântica em acervos de documentos com imagens
  • Extração automática de dados estruturados de PDFs
  • Construção de índices vetorizados para Q&A em documentos

Veredito

Solução robusta para pipeline de processamento de documentos em escala, mas exige familiaridade com stack AWS (Bedrock, OpenSearch, SQS, Batch). Bem estruturada com IaC (Terraform) e CI/CD, ideal para casos de busca documental alimentada por IA.

README

A tradução para o português deste README ainda está na fila.

Doc Search LLM to OpenSearch (AI_AGENT Architecture)

Packages

brew install poppler

Pipeline Yaml

On push request events, this workflow will run:

terraform init, terraform fmt, and terraform validate.

On push events to the "main" branch, terraform plan and terraform apply will be executed.

Documentation for hashicorp/setup-terraform is located here: https://github.com/hashicorp/setup-terraform

Resources Message Size Maximums

  1. SQS supports messages up to 256 KB in size. For larger payloads, you are charged an extra request for each additional 64 KB.

Project Knowledge ###:

  1. GH_PAT - GitHub Personal Access Token -- this is required for the GitHub Actions to run the pipline scripts. Go to GitHub -> Settings -> Developer Settings -> Personal Access Tokens -> Generate New Token -> Select Repo -> Select. -- needs done once, everyone can use the same token as long as they have access to the repo.
  2. The DLQ, topic is important, message_retention_seconds - The number of seconds Amazon SQS retains a message. Integer representing seconds, from 60 (1 minute) to 1209600 (14 days). The default for this attribute is 345600 (4 days).
  3. Turn OFF Batch Job Logs - they are not free - "Batch displays job logs from Amazon CloudWatch. With CloudWatch, there is no up-front commitment or minimum fee; you simply pay for what you use. You will be charged at the end of the month for your usage."

Repository Setup

  1. Batch Job for the file configure_opensearch_aws.py - this is the script that sets up the OpenSearch instance, it needs to be run once to set up the initial Index where all the embeddings and image summarization from Claude are stored. If you need to run it again, know that your data will be deleted and you will need to re-index all images.

  2. When you commit your work -> go to GitHub Actions to watch the pipeline run. If it fails, you can see the logs and debug from there.

  3. There are two pipelines, one for Terraform to run and a separate one for Batch Job AI_AGENT_NAME_batch.py. The Batch Job runs the AI_AGENT_NAME_batch.py script which is the main script that processes the images and sends the data to OpenSearch.

  4. If you need the Batch code to run and you haven't made any changes to the code, you can update the trigger.txt file to stimulate a change and the pipeline will run. You can also run the pipeline manually from the GitHub Actions.

  5. If you need to run Terraform locally, terraform plan -vars-file=dev.tfvars and terraform apply -vars-file=dev.tfvars make sure you include the -vars-file=dev.tfvars flag so TF knows the Workspace is either dev, stage, prod.

  6. To run the application - send folder_id and http PATH: GET/POST when calling lambda AI_AGENT_NAME_mapping. Example: event = { "body": "{"folder_id": 123}", "httpMethod": "POST" }

  7. The POST method sends SQS message to Batch via AWS Pipes. There is a DLQ on the SQS and Batch send the failed message back should it fail.

  8. The GET method retrieves data from dyanmo AI_AGENT_NAME-ENV-tracker table and returns the STATUS of processing of the pdf document, status: (IN_PROGRESS, COMPLETED, FAILED).

  9. Lambda AI_AGENT_NAME_q_a requires a POST method with the following body: { "folder_id": 123, "question": "Find images of people running", } The response is: { "folder_id": 123, "answer": "There are two pictures that have a person running. [img1.png, img2.png]", }

Ver o original no GitHub

Bom para

Relacionados