critical-thinking
Johna2an/critical-thinking
Claude Code skill that makes Claude reason with explicit probabilities, base rates, and falsifiers. Distilled from 51 books on judgment, blind-evaled against GPT-5.5, full eval transcripts included.
Instalação
npx convez-ai Johna2an/critical-thinking1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
README
A tradução para o português deste README ainda está na fila.
critical-thinking
A Claude Code skill distilled from 51 books on judgment, blind-evaled against GPT-5.5, with the full eval harness and every transcript that shaped it.
Install · What it does · Results · How it works · The story · Contributing
v2.0.0 · last updated June 2026 · changelog
critical-thinking is an open-source Agent Skill for Claude Code and Claude-based agents that makes Claude reason with explicit probabilities, base rates, and falsifiers: in blind evaluations it ranked first in 18 of 18 judge passes against baseline Claude and GPT-5.5. Frontier models already know the literature of good judgment from pretraining. Left to their defaults, they hedge in prose: "this could be confounding, or reverse causation, and the evidence is uncertain." This skill converts that latent knowledge into executed behavior. With it loaded, Claude states calibrated probabilities, anchors on base rates before vivid specifics, attaches a falsifier to every verdict, names its own cognitive traps while reasoning, and runs premortems on its own recommendations.
The repo ships three things most skill repos leave out: the full eval harness (blind, multi-arm, balanced-rotation, LLM-judged), every raw transcript and judge file from three eval rounds, and the hypotheses that broke along the way. The honest parts are the useful parts.
At a glance
| What | An Agent Skill: one operating manual (SKILL.md) + 9 reference libraries + 51 book teaching files, all markdown |
| Headline result | 8.99 vs 7.92 (Claude baseline) vs 7.30 (GPT-5.5 xhigh) mean judge score; best in 18/18 blind passes |
| Evidence | 3 eval rounds with full transcripts and judge JSON in evals/ |
| Install | Copy one folder into ~/.claude/skills/; no build, no dependencies, no API keys |
| License | MIT |
What it does
The same problem, answered by the same model on the same day. Round 1, problem 0: an observational study (n=8,000, p<0.01) links coffee to lower depression, and a wellness brand cites it to sell extract supplements. Should you believe coffee reduces depression?
This is a single observational, likely cross-sectional association. It is consistent with a real effect, with confounding, and with reverse causation, and it cannot distinguish among them. It is a reasonable prompt for a trial, not evidence that coffee treats depression.
A reasonable outside-view prior: a fresh single observational diet-mood association reflects a real, sizable causal effect in the claimed direction maybe 15 to 25 percent of the time. [...] I put roughly 20 to 25 percent that coffee causally and meaningfully reduces depression risk in adults. I put that the supplement delivers a worthwhile antidepressant effect far lower, under 10 percent.
Both answers are correct. The left one hedges in words; the right one prices its beliefs, and earlier in the same response it grounds the prior in the vitamin E / beta-carotene / HRT replication graveyard and names the study design (Mendelian randomization) that would change its mind. A judge, blind to which arm was which:
"...uniquely supplies an outside-view base rate (the vitamin E / beta-carotene / HRT reversal graveyard) with explicit probabilities (~20-25% causal, <10% supplement, 75-80% claim overstated) plus a Mendelian-randomization falsifier"
Install
The skill is a folder of markdown. No build, no dependencies.
Claude Code (macOS / Linux):
git clone https://github.com/Johna2an/critical-thinking.git
cp -r critical-thinking/skill ~/.claude/skills/critical-thinking
Claude Code (Windows, PowerShell):
git clone https://github.com/Johna2an/critical-thinking.git
Copy-Item -Recurse critical-thinking\skill "$env:USERPROFILE\.claude\skills\critical-thinking"
That's it. The skill self-triggers on reasoning-shaped requests ("is this true", "what are the odds", "red-team this", "find the flaw", "should I", "brainstorm", "what could blindside us"), and you can invoke it explicitly by asking Claude to use the critical-thinking skill. Any agent stack that supports the Agent Skills format (SKILL.md + references/) can load it the same way.
The numbers
Three rounds, every arm blind-judged on anonymized responses, three independent judges per problem, balanced slot rotation. Full data, per-problem tables, and judge quotes: docs/EVALS.md. Raw material: evals/.
The headline result, in one quotable sentence: in a blind evaluation with three independent judges per problem, Claude with the critical-thinking skill scored 8.99 out of 10 across six reasoning problems, against 7.92 for Claude without the skill and 7.30 for GPT-5.5 at maximum reasoning effort, ranking first in 18 of 18 judge passes.
| Round | Question asked | Result |
|---|---|---|
| 1 · Reasoning (6 problems) | Does the skill change reasoning at all? | Skill 8.99 vs base 7.92 vs GPT-5.5 7.30. Skill ranked best in 18 of 18 judge passes. Biggest gap: calibration, +1.73. |
| 2 · Creativity (6 briefs) | Does it help divergent work too? | Skill 8.48 vs base 7.93 vs GPT-5.5 5.96, and the win was partial: 11/18 picks, with baseline sweeping 2 of 6 briefs. The scaffold raises the floor and can cap the ceiling. |
| 3 · v1 vs v2 (8 problems) | Do 30 more books help? | v1 8.91 vs v2 8.87: a tie at the top. v2 took the most judge best-picks on the three problems where its new material supplied the winning move; v1 kept the four problems judges decided on explicit probabilities. The first 21 books had captured nearly all the headroom. |
How it works
flowchart LR
T[Task arrives] --> S[SKILL.md<br/>always in context<br/>~20KB operating manual]
S -->|task needs depth| R[9 reference libraries<br/>load on demand]
R -->|task calls a specific book| B[51 per-book teaching files<br/>load individually]
S --> O[Response with probabilities,<br/>base rates, falsifiers]
Three layers of progressive disclosure:
SKILL.md, the operating manual, is the only layer guaranteed to be in context. Eight phases that mirror how a hard problem unfolds (frame, check yourself, argue cleanly, quantify, stress-test, think in systems, generate alternatives, decide), a NEVER list of 18 named failure modes, and the section we consider the design's core: "When the directives collide", ten scoping rules for cases where two valid directives point opposite ways (trust intuition or distrust it; quantify or stay humble at the tail; generate wide or converge).- 9 reference libraries (
references/*.md): biases, argument and evidence, forecasting and decisions, creativity, systems and complexity, uncertainty and risk, formal models, epistemology, plus the original v1 combined systems-science-and-creativity library. Loaded only when the task needs that depth. - 51 per-book teaching files (
references/books/), each restating one book's working tools as directives an agent can execute. Loaded individually, rarely.
The eval data says the manual does most of the work and the library is the long tail; see hypothesis H4 in docs/HYPOTHESES.md.
The story, including the failures
The project ran as a sequence of falsifiable bets. Short version here, full version in docs/HYPOTHESES.md and docs/PROCESS.md.
Bet 1: distillation changes behavior. Held, strongly. 21 books were each distilled by a dedicated agent into executable directives, synthesized into a master outline (10 points where independent authors converge, 5 tensions resolved by scoping), then compressed into v1. Round 1: +1.07 over baseline, 18/18 judge sweeps, and judges named the same mechanisms every time: explicit numbers, outside-view base rates, falsifiers, self-named traps.
Bet 2: one manual fits all thinking. Broke. Round 2 was designed to hunt the weak flank and found it: on divergent-thinking briefs the skill won 11/18 against baseline's 7, losing the two most-rehearsed briefs ("reinvent the chair", "reinvent the report card") 3-0. Judges praised the losing skill responses as "rigorously developed" and "systematically spread", which is the vocabulary of second place, while the unscaffolded baseline produced "the single strangest swing... that still coheres". On rehearsed creative ground, systematic method walks into the same trope basins; the wild coherent outlier wins.
Bet 3: more books, better skill. Broke on the mean, held on coverage. v2 grew the library from 21 to 51 books (9 libraries, a rebuilt generation phase). Round 3 verdict: 8.91 vs 8.87, a statistical tie, with the win profile shifted exactly to where the new books live and away from the calibration habit that scores highest everywhere. Average response length was identical (914 vs 916 words), so the cost was displaced emphasis rather than bloat. The design lesson, now a contribution rule: enforcing core moves beats enlarging the library.
What's in the repo
critical-thinking/
├── skill/ # the installable skill: SKILL.md + references/ (v2, current)
├── versions/v1/ # frozen v1 (21 books), kept for reproducible A/B evals
├── evals/
│ ├── harness/ # setup, blinding, judging aggregation, report scripts
│ ├── round-1-reasoning/ # all responses, judge JSON, verdicts, HTML report
│ ├── round-2-creativity/
│ └── round-3-v1-vs-v2/
├── docs/
│ ├── EVALS.md # full results + honest limitations
│ ├── HYPOTHESES.md # the bets, what held, what broke, why
│ ├── PROCESS.md # the build pipeline, end to end
│ ├── ROADMAP.md # ten open problems, ordered by value
│ └── master-synthesis.md # the architect's blueprint from the 21-book corpus
└── assets/ # banner, results chart (SVG)
Each eval round folder is self-auditing: open _report.html in a browser for side-by-side responses with scores, or recompute the tables yourself from the raw judge JSON and rotation maps (the aggregators are deterministic).
Limitations, before you quote the numbers
- Claude judged Claude. All judges were Claude agents; family self-preference is a documented LLM-as-judge bias. Blind rotation cancels position bias, family bias survives it. Re-judging the included blinded transcripts with another model family is the top item on the roadmap.
- n=1 per arm per problem. Large effects (Round 1) survive that; the 0.04 v1-v2 gap does not.
- Authored problems. The problem sets were written by the project. Adversarial, held-out problem sets are wanted.
Full list in docs/EVALS.md.
Contributing
The contribution rule follows from the project's own history: claims ride on evals. State the hypothesis, run the relevant round before and after, ship the diff with the data. The seven highest-value open problems, from independent re-judging to a v3 that enforces calibration as a hard rule, are tabulated in CONTRIBUTING.md and docs/ROADMAP.md.
The library
| Book | Author(s) |
|---|---|
| Thinking, Fast and Slow | Kahneman |
| The Scout Mindset | Galef |
| Superforecasting | Tetlock & Gardner |
| The Signal and the Noise | Silver |
| Thinking in Bets | Duke |
| Asking the Right Questions | Browne & Keeley |
| A Rulebook for Arguments | Weston |
| Thinking from A to Z | Warburton |
| How to Lie with Statistics | Huff |
| Calling Bullshit | Bergstrom & West |
| Thinking in Systems | Meadows |
| The Demon-Haunted World | Sagan |
| The Beginning of Infinity | Deutsch |
| Nonsense on Stilts | Pigliucci |
| Lateral Thinking | de Bono |
| Intuition Pumps | Dennett |
| Factfulness | Rosling |
| Rational Choice in an Uncertain World | Hastie & Dawes |
| Judgment in Managerial Decision Making | Bazerman & Moore |
| How to Read a Book | Adler & Van Doren |
| The Art of Doing Science and Engineering | Hamming |
The Act of Creation (Koestler) · Six Thinking Hats (de Bono) · A Whack on the Side of the Head (von Oech) · Where Good Ideas Come From (Johnson) · Change by Design (Brown) · Creative Confidence (Kelley & Kelley) · The Creative Habit (Tharp) · The Creative Act (Rubin) · Lessons in Creativity · Designing Your Life (Burnett & Evans) · The Design of Everyday Things (Norman) · Surfaces and Essences (Hofstadter & Sander) · Gödel, Escher, Bach (Hofstadter) · The Mind's I (Hofstadter & Dennett) · The Society of Mind (Minsky)
Systems: Seeing Like a State (Scott) · Systemantics (Gall) · The Fifth Discipline (Senge) · The Death and Life of Great American Cities (Jacobs)
Uncertainty and risk: Antifragile, The Black Swan, Fooled by Randomness (Taleb)
Models and strategy: The Model Thinker (Page) · An Introduction to Statistical Learning (James, Witten, Hastie, Tibshirani) · Models of My Life (Simon) · The Art of Strategy (Dixit & Nalebuff)
Epistemology: The Structure of Scientific Revolutions (Kuhn) · The Logic of Scientific Discovery (Popper) · Against Method (Feyerabend) · What is this thing called Knowledge? (Pritchard)
The reference files are original prose: each book's ideas and methods restated as directives, written from scratch, with no reproduced text. Raw book text never enters this repo. If you are an author or publisher with a concern about a teaching file, open an issue and we will address it directly.
FAQ
What is the critical-thinking skill? An open-source Agent Skill for Claude Code: a folder of markdown (an operating manual plus reference libraries) that loads into Claude's context and changes how it reasons. It was distilled from 51 books on judgment, decision-making, forecasting, systems thinking, creativity, and the philosophy of science.
How is this different from telling Claude to "think carefully"? The skill replaces vague instruction with named, executable moves: state a calibrated probability, anchor on a base rate before case specifics, attach a falsifier to every verdict, run a premortem. In blind evals, those specific behaviors produced the entire scoring gap over a "reason carefully" baseline prompt that both arms shared.
How much does it improve Claude's reasoning? On a 6-problem reasoning battery scored 0 to 10 by three blind judges per problem, Claude with the skill scored 8.99 against 7.92 without it and 7.30 for GPT-5.5 at xhigh reasoning effort, taking first place in 18 of 18 judge passes. Read docs/EVALS.md for the limitations before quoting these numbers, starting with the fact that the judges were Claude agents.
Does it work with ChatGPT, GPT, Gemini, or local models? Untested, and testing it is roadmap item 3. SKILL.md is plain markdown with no Claude-specific machinery, so it can be pasted into any system prompt. If the behavioral shifts transfer, this becomes a portable reasoning protocol; results from other models are a welcome contribution.
Is it free?
Yes. MIT license, no build step, no dependencies, no API keys. Copy one folder into ~/.claude/skills/.
Where does the skill's knowledge come from? Is book text included?
Every reference file is original prose: each book's methods restated as directives, written from scratch. The repo distributes no book text, and raw sources are excluded by .gitignore pattern.
Can I add a book or change a directive? Yes, and the bar is empirical: state the hypothesis, run the relevant eval round before and after, ship the diff with the data. CONTRIBUTING.md has the loop, plus the seven open problems where help is most wanted.
License
MIT. The eval transcripts and judge outputs are released under the same license; cite via CITATION.cff if you build on the results.