convez.ai
MCP ServerDevOps & InfraestruturaAvançado
ravisharma5

OGX/LlamaStack no RHOAI com MCP

ravisharma5/rhoai-llamastack

Deploy OGX+MCP em RHOAI com vLLM e PostgreSQL

Ver no GitHub
InicialPluginScore · de 0 a 100

MCP server para deploy de OGX (antigo LlamaStack) em OpenShift AI 3.4, integrando endpoints vLLM remotos, PostgreSQL e múltiplos provedores de MCP. Automatiza a orquestração de inferência, embeddings e armazenamento em Kubernetes.

Instalação

$npx convez-ai ravisharma5/rhoai-llamastack

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
1
Acompanhando
0
Saúde
28%
Último commit
há 5 meses
Versão
—
Issues abertas
0
Contribuidores
1

Casos de uso

  • Implantar servidores LLM em clusters OpenShift gerenciados
  • Integrar múltiplos endpoints vLLM (inference e embedding) centralizadamente
  • Orquestrar MCP servers para observabilidade (VictoriaMetrics, detecção de incidentes)

Veredito

Solução especializada para quem opera RHOAI em produção e precisa deployer OGX com MCP servers. Requer conhecimento firme de OpenShift, vLLM e containerização; documentação é técnica e bem estruturada, mas sem abstrações simplificadas para iniciantes.

README

A tradução para o português deste README ainda está na fila.

OGX (formerly LlamaStack) on RHOAI 3.4

Deploy an OGX server with MCP server integration on OpenShift AI 3.4 using the LlamaStackDistribution custom resource.

Migration note: OGX v0.7.1 ships with RHOAI 3.4, replacing LlamaStack v0.4.x from RHOAI 3.3. See Migration from RHOAI 3.3 for breaking changes.

Architecture

External vLLM Endpoints
  (Inference + Embedding)
         |
         v
+---------------------+         +-------------------------+
| OGX Server          |--SSE--->| OpenShift MCP Server    |
| (port 8321)         |         | (port 8080)             |
|                     |         +-------------------------+
| Providers:          |
|  - vllm-inference   |         +-------------------------+
|    (remote::vllm)   |--SSE--->| VictoriaMetrics MCP     |
|  - vllm-embedding   |         | Aggregator (port 8000)  |
|    (remote::vllm)   |         +-------------------------+
|  - inline::builtin  |
|    (responses)      |         +-------------------------+
|  - model-context-   |--SSE--->| Incident Detection MCP  |
|    protocol         |         | (port 8085)             |
|                     |         +-------------------------+
| APIs:               |
|  - inference         |
|  - responses         |
|  - tool_runtime      |
|  - vector_io         |
|  - files             |
+---------------------+
         |
         v
+---------------------+
| PostgreSQL          |
| (KV + SQL storage)  |
+---------------------+

Prerequisites

  • OpenShift AI 3.4 with the LlamaStack/OGX Operator activated (llamastackoperator.managementState: Managed in your DataScienceCluster CR)
  • oc CLI logged in to your cluster
  • envsubst available (part of gettext)
  • An external vLLM inference endpoint URL (ending in /v1) and API token
  • An external vLLM embedding endpoint URL and API token

Quick Start

# 1. Clone the repo
git clone https://github.com/ravisharma5/rhoai-llamastack.git
cd rhoai-llamastack

# 2. Configure your environment
cp .env.example .env
# Edit .env with your vLLM endpoints and API tokens

# 3. Login to your OpenShift cluster
oc login --server=https://api.your-cluster.com:6443

# 4. Deploy everything
./deploy.sh

# 5. Verify
oc get pods
OGX_URL=https://$(oc get route llama-stack-server -o jsonpath='{.spec.host}')
curl -sk $OGX_URL/v1/health
curl -sk $OGX_URL/v1/models

Step-by-Step Deployment

1. Create namespace

oc new-project llama-stack

2. Configure and create secrets

cp .env.example .env
# Edit .env with your values:
#   VLLM_URL, VLLM_API_TOKEN
#   VLLM_EMBEDDING_URL, VLLM_EMBEDDING_API_TOKEN
#   POSTGRES_PASSWORD
source .env
envsubst < llama-stack-secret.yaml | oc apply -f -

3. Create OGX server config

The config uses OGX v0.7.x schema. Key differences from v0.4.x:

version: 2
distro_name: rh                    # was: image_name

apis:
  - inference
  - responses                      # was: agents
  - tool_runtime
  - vector_io
  - files
  # Removed: eval, scoring, datasetio, safety

providers:
  inference:
    - provider_id: vllm-inference
      provider_type: remote::vllm
      config:
        base_url: ${env.VLLM_URL}  # must be base_url, not url
        max_tokens: 8192
        api_token: ${env.VLLM_API_TOKEN}
        network:                   # was: tls_verify (flat field)
          tls:
            verify: false

  responses:                       # was: agents
    - provider_id: builtin         # was: meta-reference
      provider_type: inline::builtin  # was: inline::meta-reference
      config:
        persistence:
          responses:
            backend: sql_default
            table_name: agents_responses

  tool_runtime:
    - provider_id: model-context-protocol
      provider_type: remote::model-context-protocol
      config: {}

Apply the ConfigMap:

oc create configmap llama-stack-run-config \
  --from-file=config.yaml=llamastack-config.yaml

Important: The ConfigMap key must be config.yaml. The operator mounts it at /etc/llama-stack/config.yaml.

4. Deploy PostgreSQL

oc apply -f postgres.yaml
oc wait --for=condition=available deployment/postgres-llamastack --timeout=120s

5. Deploy OpenShift MCP Server

oc apply -f openshift-mcp-server.yaml
oc wait --for=condition=available deployment/openshift-mcp-server --timeout=120s

6. Deploy OGX Server (LlamaStackDistribution CR)

The CR tells the RHOAI operator to deploy and manage the OGX server:

apiVersion: llamastack.rhoai.opendatahub.io/v1alpha1
kind: LlamaStackDistribution
metadata:
  name: llama-stack-server
spec:
  replicas: 1
  userConfig:                      # must be userConfig, not userConfigMapRef
    configMapName: llama-stack-run-config
  secretRef:
    name: llama-stack-secret
  network:
    allowedFrom:
      namespaces: ["*"]            # allows OpenShift router traffic
oc apply -f llamastackdistribution.yaml

7. Create external route

oc create route edge llama-stack-server \
  --service=llama-stack-server-service --port=8321

8. Verify deployment

# Check pods
oc get pods

# Follow logs
oc logs -l app.kubernetes.io/instance=llama-stack-server -f

# Health check
OGX_URL=https://$(oc get route llama-stack-server -o jsonpath='{.spec.host}')
curl -sk $OGX_URL/v1/health

# List models
curl -sk $OGX_URL/v1/models

# List registered toolgroups (still works in v0.7.x)
curl -sk $OGX_URL/v1/toolgroups

Responses API

The /v1/responses API handles the full agent loop server-side: tool discovery, LLM-driven tool selection, execution, and response generation.

Basic query

from llama_stack_client import LlamaStackClient

client = LlamaStackClient(base_url="https://<ogx-route-url>")

response = client.responses.create(
    model="vllm-inference/your-model-id",
    input="List all pods in the llama-stack namespace",
    tools=[
        {
            "type": "mcp",
            "server_label": "openshift",
            "server_url": "http://openshift-mcp-server:8080/sse",
            "require_approval": "never",
        }
    ],
    stream=False,
)

New parameters in v0.7.x

response = client.responses.create(
    model="vllm-inference/your-model-id",
    input="Investigate high CPU usage across all namespaces",
    tools=mcp_tools,
    stream=True,
    instructions="You are an AIOps assistant...",
    # New in v0.7.x:
    parallel_tool_calls=True,       # concurrent MCP queries
    max_output_tokens=4096,         # per-request output limit
    max_tool_calls=10,              # total tool call cap
    reasoning={"effort": "medium"}, # reasoning depth control
    truncation="auto",              # was: extra_body={"truncation": "auto"}
)
ParameterPurposeDefault
parallel_tool_callsAllow model to emit multiple tool calls per turnTrue
max_output_tokensPer-request output token limitModel default
max_tool_callsCap total tool invocations per requestNo limit
reasoningControl reasoning depth (none/minimal/low/medium/high/xhigh)None
truncationContext window management strategyNone

Streaming events

for event in response:
    if event.type == "response.output_text.delta":
        print(event.delta, end="")
    elif event.type == "response.output_item.added":
        if event.item.type == "mcp_call":
            print(f"Calling: {event.item.name}")
    elif event.type == "response.output_item.done":
        if event.item.type == "mcp_call":
            print(f"Result: {event.item.output[:200]}")
    elif event.type == "response.completed":
        print("Done")
    elif event.type == "response.failed":
        print(f"Error: {event.response.error}")

MCP Server Configuration

Registering MCP toolgroups

MCP servers are registered in the OGX config under registered_resources.tool_groups:

registered_resources:
  tool_groups:
    - toolgroup_id: mcp::openshift
      provider_id: model-context-protocol
      mcp_endpoint:
        uri: http://openshift-mcp-server:8080/sse

    - toolgroup_id: mcp::victoriametrics
      provider_id: model-context-protocol
      mcp_endpoint:
        uri: http://victoriametrics-mcp-aggregator:8000/sse

Safety modes

ModeFlagWhat it allows
Read-only--read-onlyList, get, describe only
Non-destructive--disable-destructiveRead + create (no delete/update)
Full access(no flag)All operations

Authentication (OAuth2/Keycloak)

OGX supports OAuth2 token authentication via Keycloak:

server:
  port: 8321
  auth:
    provider_config:
      type: "oauth2_token"
      jwks:
        uri: "http://keycloak-service.rhbk.svc:8080/realms/aiops/protocol/openid-connect/certs"
      issuer: "https://keycloak-route/realms/aiops"
      audience: "account"

Client usage with auth:

client = LlamaStackClient(
    base_url="https://<ogx-route-url>",
    api_key="<jwt-token>",
)

Migration from RHOAI 3.3

Breaking changes (v0.4.x → v0.7.x)

AreaOld (v0.4.x)New (v0.7.x)
Config fieldimage_name: rhdistro_name: rh
Agents providerinline::meta-referenceinline::builtin
Agents APIagents in apis listresponses in apis list
TLS configtls_verify: falsenetwork: { tls: { verify: false }}
RAG toolgroupbuiltin::ragbuiltin::file_search
SDK versionllama-stack-client>=0.4.0,<0.5llama-stack-client>=0.7.0,<0.8
Removed APIs—eval, scoring, datasetio, safety
Removed section—registered_resources.tool_groups auto-registers

SDK changes

# Import path is UNCHANGED
from llama_stack_client import LlamaStackClient

# New parameters available on responses.create():
# parallel_tool_calls, max_output_tokens, max_tool_calls, reasoning, truncation

# max_infer_iters replaced by max_tool_calls
# extra_body={"truncation": "auto"} replaced by truncation="auto"

Adding a Remote OpenShift Cluster

See the remote cluster setup guide in the previous version of this README. The process is unchanged — deploy a second MCP server pod with a kubeconfig pointing to the remote cluster, register it as mcp::openshift-remote in the OGX config.

Updating Configuration

oc create configmap llama-stack-run-config \
  --from-file=config.yaml=llamastack-config.yaml \
  --dry-run=client -o yaml | oc apply -f -

# Restart to pick up changes
oc rollout restart deployment/llama-stack-server

Teardown

./teardown.sh
# Or manually:
oc delete llamastackdistribution llama-stack-server
oc delete route llama-stack-server
oc delete configmap llama-stack-run-config
oc delete -f openshift-mcp-server.yaml
oc delete -f postgres.yaml
envsubst < llama-stack-secret.yaml | oc delete -f -

Troubleshooting

OGX pod in CrashLoopBackOff:

  • Check logs: oc logs -l app.kubernetes.io/instance=llama-stack-server
  • Verify ConfigMap key is config.yaml (not run.yaml)
  • Verify base_url (not url) in vLLM provider config
  • Verify distro_name (not image_name)
  • Verify inline::builtin (not inline::meta-reference)

Route returns 503:

  • Check NetworkPolicy: oc get networkpolicy -o yaml
  • Ensure network.allowedFrom.namespaces: ["*"] in the CR

MCP tools not appearing:

  • Verify toolgroup entries in config under registered_resources.tool_groups
  • Verify model-context-protocol provider under providers.tool_runtime
  • Check OGX logs for MCP connection errors

Auth errors (401/403):

  • Verify Keycloak JWKS URI is reachable from OGX pod
  • Verify issuer matches the external Keycloak route URL (JWT iss claim uses the external URL)
  • Check token expiry: python3 -c "import jwt; print(jwt.decode(token, options={'verify_signature':False}))"

Silent accuracy regression after upgrade:

  • Upgrade OGX and vLLM atomically — Red Hat benchmarks show accuracy drops when only one is upgraded
  • Run test queries before and after to verify quality
Ver o original no GitHub

Bom para

Relacionados