convez.ai
PluginDados & AnalyticsAvançado
antonpstahl

XAI para Previsões ML com Explicações em Linguagem Natural

antonpstahl/Implementation-XAI-Stahl-ss26

Estude como LLMs geram melhores explicações variando XAI methods e formatos

Ver no GitHub
InicialPluginScore · de 0 a 100

Compara métodos de interpretabilidade (EBM vs. SHAP) e formatos de entrada (JSON, PNG, tool-use) para gerar explicações em linguagem natural via LLM sobre previsões de modelos ML. Projeto acadêmico da TU Dresden que estuda como LLMs traduzem predições em explanações acessíveis a usuários não-técnicos, usando o dataset Capital Bikeshare como case.

Instalação

$npx convez-ai antonpstahl/Implementation-XAI-Stahl-ss26

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
0
Forks
0
Acompanhando
0
Saúde
28%
Último commit
há 1 mês
Versão
—
Issues abertas
0
Contribuidores
2

Casos de uso

  • Gerar explicações automáticas de previsões ML para stakeholders não-técnicos
  • Comparar qualidade de explicações entre métodos interpretáveis (EBM) e post-hoc (SHAP)
  • Testar diferentes formatos de handover (JSON, imagens, tool-use) para LLMs

Veredito

Projeto rigoroso e bem-estruturado que endereça uma questão prática relevante: como apresentar explicações de ML a não-especialistas via LLM. O código é reproduzível e documentado, mas requer familiaridade com XAI, XGBoost, EBM e APIs Claude para aproveitar plenamente. Ideal como referência pedagógica ou base para sistemas de explicação interpretável em produção.

README

A tradução para o português deste README ainda está na fila.

LLM-Assisted XAI Explanations for ML Predictions

Comparing how the choice of XAI method and the handover format to a large language model (LLM) affect the quality of automatically generated, natural-language explanations of ML predictions.

This repository accompanies a term project (Studienarbeit) at TU Dresden, supervised by Prof. Dr. Patrick Zschech (Chair of Business Information Systems, esp. Intelligent Systems and Services).

Overview

The project studies how LLMs can translate predictions from machine-learning models into natural-language explanations for non-expert end users. The application case is the Capital Bikeshare system in Washington, D.C. — an hourly bike-rental demand dataset.

Two questions are examined in parallel:

  1. XAI method — do explanations grounded in an inherently interpretable model (EBM shape functions) beat post-hoc explanations (SHAP on XGBoost)?
  2. Handover format — does the LLM produce better explanations when it receives the information as structured JSON, as an image (waterfall plot, PNG), or through active tool calls (tool-use)?

Repository structure

.
├── data/            # Raw data and prepared train/test splits
├── models/          # Trained models (6 .pkl files)
├── explanations/    # SHAP / EBM explanations as JSON + waterfall plots (PNG)
├── results/         # Pipeline outputs, evaluation plots, CSV summaries
├── notebooks/       # 10 Jupyter notebooks (00 baseline, 01–08)
├── prompts/         # Prompt templates
└── utils/           # Python helper modules (data, models, explanations, llm, tools)

Pipeline

1 — Data preprocessing (01_Data_Preprocessing.ipynb) UCI Bike Sharing dataset (17,379 hourly observations, 2011–2012). Leakage and redundant features removed; multicollinearity handled (atemp vs. temp, r ≈ 0.99); categorical encoding for native splits; log1p target transform; 70/30 train/test split. Nine features remain (hr, mnth, weekday, weathersit, yr, holiday, temp, hum, windspeed).

2 — Modeling (02a_Modeling_AllOptions.ipynb, 02b_Comparison.ipynb) XGBoost and EBM (InterpretML), each trained with three loss functions. Poisson-log was selected for all downstream steps (best Poisson deviance, no negative predictions).

LossModelRMSEMAER²Poisson dev.Neg. pred.
Poisson-logXGB45.4427.000.9359.380
Poisson-logEBM48.2028.200.92710.810

(Test set n = 5,227; values from results/model_metrics_poisson_log.json.)

3 — Explanation generation (03_Explanations_Generation.ipynb) Global explanations (SHAP feature importance for XGB; term importances for EBM) and local explanations for 10 test instances stratified across cnt quintiles, stored as JSON plus waterfall-plot PNGs.

4 — Three LLM pipelines + a deterministic baseline All LLM pipelines use claude-sonnet-4-6 and produce three-part explanations ([PREDICTION], [DRIVERS], [RECOMMENDATION]) for non-technical staff.

  • 00 Template (baseline) — a deterministic text-block generator that fills the same three-part structure from the identical SHAP/EBM JSON, with no LLM call. Answers the standard reviewer question: what does the LLM add over a template?
  • 04 JSON → Text — the LLM receives global importance and local SHAP/EBM contributions as structured JSON. System prompt cached via Anthropic prompt caching; raw values denormalized into plain language (e.g. temp=0.68 → ~27.9 °C) before the call.
  • 05 Vision → Text — the LLM receives the instance's waterfall plot as a base64-encoded PNG and reads bar lengths visually (no numeric access to contribution values).
  • 06 Tool-Use — the LLM retrieves data itself through 8 defined tools (feature schema, importance, prediction, SHAP values, partial dependence, value context, similar instances, counterfactuals) in an agentic loop — averaging 5.65 tool calls per explanation.

5 — Evaluation (07_Evaluation.ipynb, 08_Evaluation_Ichmoukhamedov.ipynb) Quantitative cost/latency, LLM-as-judge across three judge versions (uncalibrated Sonnet, calibrated-rubric Sonnet, independent Opus), and formal faithfulness metrics after Ichmoukhamedov et al. (Rank / Sign / Value Agreement).

Quantitative + LLM-judge (v1) summary across 20 explanations per pipeline (2 XAI models × 10 instances):

PipelineAvg wordsInput tok.¹Output tok.Cost (20 calls)Avg latencyJudge Faith.ClarityComplete.
Template5400$0.000.0 s5.004.704.00
JSON→Text208616510$0.1611.7 s4.354.904.95
Vision2122,167528$0.2912.3 s3.804.554.75
Tool-Use3053,4891,225$0.5828.8 s4.403.954.90

¹ Input tokens are the billed, non-cached count. JSON→Text caches the system prompt (cache-read tokens, billed at ~10%, are not counted here), which is why its input count is far below Vision's freshly-sent image tokens. Values from results/eval_summary.csv.

Formal faithfulness after Ichmoukhamedov et al. (NB 08, n = 10 instances; precision-style metrics — see limitation in NB 08 §4.1):

PipelineRank Agr.Sign Agr.Value Agr.
JSON→Text0.5620.7210.667
Tool-Use0.5580.7330.733
Vision0.4290.6790.575

Status of these findings: descriptive/exploratory. With n = 10–20 explanations per pipeline, no repeated sampling and no inferential statistics yet, the differences below are not statistically confirmed (see the limitations table in 07_Evaluation.ipynb §7). Treat them as directional.

  1. The deterministic template wins on faithfulness — Template scores 5.00 vs. the LLM pipelines' 3.80–4.40. By construction it lists exactly the true top drivers; the LLMs trade some faithfulness for richer, more readable narratives. This is the central "what does the LLM add over a template?" result — and the trade-off, not a free lunch.
  2. Among LLM pipelines, faithfulness ranks Tool-Use (4.40) ≈ JSON→Text (4.35) > Vision (3.80) — consistent with the formal Rank-Agreement (Vision 0.43 vs. ~0.56 for the others): reading bar lengths from a plot is structurally less precise than numeric access.
  3. Clarity and Completeness are at the ceiling for the LLM pipelines (≥ 4.55 / ≥ 4.75) and do not discriminate between them; the template lags on completeness (4.00, thinner operational recommendation).
  4. JSON→Text is most efficient (≈ $0.008 per explanation, lowest latency 11.7 s) — system-prompt caching keeps billed input tokens low.
  5. Tool-Use produces the longest, evidence-backed explanations (+47% words vs. JSON→Text, with partial-dependence and counterfactual support) at ~3.6× cost and ~2.5× latency.
  6. Vision matches JSON→Text on latency but costs more (image tokens, no caching benefit) and has the lowest faithfulness of all pipelines.
  7. Possible self-preference bias — Opus (v3) scores sit systematically below Sonnet (v1/v2) under an identical rubric. This is consistent with a self-preference effect but not conclusive: both judges are Anthropic models, so a true cross-vendor judge is still outstanding (see implementation plan, Phase 2).

Setup

# dependencies (Python 3.13.1)
pip install -r requirements.txt

# API key (do NOT commit your .env)
echo "ANTHROPIC_API_KEY=sk-ant-..." > .env

Run the notebooks in order (01 → 08). Paths are relative to the project root; reproducibility is fixed via RANDOM_STATE = 42.

LLM configuration

All LLM calls use the Anthropic Messages API (accessed 2026-06-11). Parameters are centralised in utils/llm.py.

Use caseModelmax_tokenstemperature
Explanation generation (NB 04 / 05 / 06)claude-sonnet-4-62048default (1.0)
Faithfulness check (NB 07)claude-sonnet-4-6300default (1.0)
Judge v1 uncalibrated (NB 07)claude-sonnet-4-6600default (1.0)
Judge v2 calibrated (NB 07)claude-sonnet-4-6600default (1.0)
Judge v3 independent (NB 07)claude-opus-4-8600default (1.0)
Ichmoukhamedov metrics (NB 08)claude-sonnet-4-6700default (1.0)

Reproducibility note (→ Paper limitation): Anthropic model IDs are versioned snapshots, but API behaviour (sampling, default parameters, tokenisation) can change silently between SDK releases. Results are tied to anthropic==0.98.1 and the access date above. Future runs against the same model ID are not guaranteed to produce identical outputs.

Test suite

pytest tests/        # run all tests
pytest tests/test_prompt_golden.py -v   # prompt regression only

The suite covers sampling determinism, generation-loop persistence/resume, judge-parsing robustness, statistical functions, denormalization consistency, README consistency, and prompt-fix regression. The prompt regression test (test_prompt_golden.py) freezes the SHA-256 hashes and key constraint phrases of all three pipeline prompts as corrected in Phase 3 (sign- and rank-fidelity rules for yr=0). It is a hard gate: Phase 3b (full-scale run) must not start until all tests are green.

Green-gate status (Phase 3a): pytest tests/ → 139 passed (2026-06-17, Python 3.13). All scaling-critical paths are covered, so Phase 3b is released.

If a prompt is intentionally improved:

  1. Edit the prompt file.
  2. Recompute the hash: shasum -a 256 prompts/<file>.md
  3. Update GOLDEN_HASHES in tests/test_prompt_golden.py.
  4. If the constraint phrase changed, update REQUIRED_PHRASES in the same file.
  5. Confirm pytest tests/test_prompt_golden.py passes.

Notes

  • The .env file and any API keys are excluded from version control and must be supplied locally.
  • This repository contains the author's own code, data preparation, and results. Third-party publications are not redistributed here.

Context

Term project (Studienarbeit), Information Systems, TU Dresden — supervised by Prof. Dr. Patrick Zschech. A follow-up Diplom thesis (master's-thesis equivalent) extends this work.

Ver o original no GitHub

Bom para

Relacionados