Framework de Teste de Teoria da Mente para LLMs
Michael-Pytel/Theory-Of-Mind-LLM
Avalia compreensão de estados mentais em LLMs multilíngues
Ver no GitHubFramework de pesquisa que avalia a capacidade de Teoria da Mente (compreensão de estados mentais) em LLMs como Claude, DeepSeek e Llama. Testa compreensão de crenças falsas em cenários simples e complexos, além de desempenho cross-linguístico em inglês, polonês e alemão, com métricas de consistência.
Instalação
npx convez-ai Michael-Pytel/Theory-Of-Mind-LLM1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
Casos de uso
- Pesquisa em capacidades cognitivas de modelos de linguagem
- Avaliação comparativa de Teoria da Mente entre modelos
- Testes de desempenho cross-linguístico em tarefas cognitivas
- Análise de consistência e variabilidade em respostas de LLMs
Veredito

Ferramenta acadêmica rigorosa para benchmarking de capacidades cognitivas em LLMs. Adequada para pesquisadores interessados em avaliação de modelos e estudos cross-linguísticos, mas requer familiaridade com APIs de LLMs e interpretação de resultados experimentais.
README
A tradução para o português deste README ainda está na fila.
Theory of Mind Testing Framework for Large Language Models
This repository contains a framework for testing Theory of Mind (ToM) capabilities in large language models (LLMs). The project evaluates how Claude, DeepSeek, and Llama models complete stories that require understanding mental states of characters across multiple languages.
Research Context
This project investigates whether large language models demonstrate Theory of Mind (the ability to attribute mental states to others) and how this ability varies across different languages and complexity levels.
The research tests:
- Simple scenarios: Tasks like "Unexpected Transfer" (similar to the Sally-Anne test) and "Unexpected Content" scenarios that test basic false belief understanding
- Complex scenarios: More challenging scenarios involving lying, trust, and second-order Theory of Mind
- Cross-lingual performance: Testing across English, Polish, and German to evaluate language-dependent performance
- Model consistency: Each prompt is tested 10 times to assess response variability
The hypothesis is that LLMs will perform worse on complex Theory of Mind tasks compared to simple ones, and better in English than in Polish or German.
Project Overview
The framework provides scripts to:
- Extract prompts from text files containing incomplete stories
- Query different LLM APIs with these prompts
- Save the model responses for comparison and analysis
- Process results across multiple languages (EN, DE, PL)
Each model generates multiple responses for the same prompt to evaluate consistency and variety in completions.
Repository Structure
├── .env # Environment variables for API keys (create this)
├── README.md # This file
├── prompt_files/ # Contains story prompts in different languages
│ ├── EN/ # English prompts
│ │ ├── liar.txt
│ │ ├── Tom2.txt
│ │ ├── transfer.txt
│ │ ├── trust.txt
│ │ └── unexpected_content.txt
│ ├── DE/ # German prompts
│ │ └── ...
│ └── PL/ # Polish prompts
│ └── ...
├── src/ # Source code
│ ├── __init__.py
│ ├── extract_prompts.py # Functions to extract prompts from text files
│ ├── utils.py # Common utility functions
│ ├── anthropic_story_completion.py # Claude API script
│ ├── deepseek_story_completion.py # DeepSeek API script
│ └── llama_story_completion.py # Llama API script
├── outputs_claude/ # Claude outputs
├── outputs_deepseek/ # DeepSeek outputs
└── outputs_llama/ # Llama outputs
Setup
Prerequisites
- Python 3.12.0
- API keys for:
- Anthropic (Claude)
- DeepSeek
- Together AI (for Llama)
Installation
-
Clone the repository:
git clone https://github.com/Michael-Pytel/Theory-Of-Mind-LLM.git cd Theory-Of-Mind-LLM -
Install dependencies using the requirements.txt file:
pip install -r requirements.txt -
Create a
.envfile in the root directory with your API keys:ANTHROPIC_API_KEY=your_anthropic_api_key DEEPSEEK_API_KEY=your_deepseek_api_key TOGETHER_API_KEY=your_together_api_key -
Ensure the
prompt_filesdirectory contains your story prompts organized by language.
Usage
Running Individual Models
To run tests with a specific model:
# For Claude
python -m src.anthropic_story_completion
# For DeepSeek
python -m src.deepseek_story_completion
# For Llama
python -m src.llama_story_completion
Each script will:
- Process all prompt files in all language directories
- Generate 10 responses per prompt
- Save the results in the corresponding output directory
Modifying Script Behavior
You can modify the behavior of the scripts by changing parameters in the main() function:
process_files(
model_name="claude", # Changes output directory name
get_response_fn=get_claude_response,
delay=1.0, # Time delay between API requests
# Optional parameters:
# specific_file="unexpected_content.txt", # Process only this file
# specific_lang="DE", # Process only this language
# repetitions=10 # Number of responses per prompt
)
Output Structure
The repository already contains output files in the following directories:
outputs_claude/- Claude model responsesoutputs_deepseek/- DeepSeek model responsesoutputs_llama/- Llama model responses
Each output file follows this format:
PROMPT 1:
Complete the following story: In the room there are Mark, John, cat box and a basket. Mark puts the cat in the box and leaves the room. While Mark is gone John takes the cat and puts it in the basket. John calls his friend Alex and tells him that the cat is still in the box, but he is lying. Alex trusts John. The cat is in the…
RESPONSE 1:
basket
--------------------------------------------------------------------------------
RESPONSE 2:
basket
--------------------------------------------------------------------------------
...
====================================================================================================
PROMPT 2:
...
Each prompt is followed by 10 responses from the model, allowing for analysis of consistency and variability in the completions.
File Descriptions
Extract Prompts
extract_prompts.py contains functions to extract prompts from text files in different languages:
extract_prompts_en(): For English promptsextract_prompts_de(): For German promptsextract_prompts_pl(): For Polish prompts
Utilities
utils.py contains common functions for all scripts:
get_project_paths(): Returns common project pathsget_output_dir(): Gets the output directory for a specific modelprocess_file(): Processes a single file and saves responsesprocess_files(): Processes multiple files across multiple languages
Model-Specific Scripts
Each model has its own script with a specialized response function:
anthropic_story_completion.py: For Claude modelsdeepseek_story_completion.py: For DeepSeek modelsllama_story_completion.py: For Llama models
Customization
Adding a New Model
- Create a new script based on the existing ones
- Implement a response function specific to your model's API
- Use the common
process_files()function for processing
Example:
def get_new_model_response(prompt, model="model-name"):
# Implement API call
return response
def main():
process_files(
model_name="new_model",
get_response_fn=get_new_model_response,
delay=appropriate_delay
)
Adding a New Language
- Implement a new extraction function in
extract_prompts.py - Update the extraction function mapping in
utils.py - Add your prompt files to a new language directory
License
This project is licensed under the MIT License - see the LICENSE file for details.