convez.ai
PluginDados & AnalyticsAvançado
Michael-Pytel

Framework de Teste de Teoria da Mente para LLMs

Michael-Pytel/Theory-Of-Mind-LLM

Avalia compreensão de estados mentais em LLMs multilíngues

Ver no GitHub
InicialPluginScore · de 0 a 100

Framework de pesquisa que avalia a capacidade de Teoria da Mente (compreensão de estados mentais) em LLMs como Claude, DeepSeek e Llama. Testa compreensão de crenças falsas em cenários simples e complexos, além de desempenho cross-linguístico em inglês, polonês e alemão, com métricas de consistência.

Instalação

$npx convez-ai Michael-Pytel/Theory-Of-Mind-LLM

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
1
Forks
0
Acompanhando
2
Saúde
42%
Último commit
há 1 ano
Versão
—
Issues abertas
0
Contribuidores
2

Casos de uso

  • Pesquisa em capacidades cognitivas de modelos de linguagem
  • Avaliação comparativa de Teoria da Mente entre modelos
  • Testes de desempenho cross-linguístico em tarefas cognitivas
  • Análise de consistência e variabilidade em respostas de LLMs

Veredito

Ferramenta acadêmica rigorosa para benchmarking de capacidades cognitivas em LLMs. Adequada para pesquisadores interessados em avaliação de modelos e estudos cross-linguísticos, mas requer familiaridade com APIs de LLMs e interpretação de resultados experimentais.

README

A tradução para o português deste README ainda está na fila.

Theory of Mind Testing Framework for Large Language Models

This repository contains a framework for testing Theory of Mind (ToM) capabilities in large language models (LLMs). The project evaluates how Claude, DeepSeek, and Llama models complete stories that require understanding mental states of characters across multiple languages.

Research Context

This project investigates whether large language models demonstrate Theory of Mind (the ability to attribute mental states to others) and how this ability varies across different languages and complexity levels.

The research tests:

  • Simple scenarios: Tasks like "Unexpected Transfer" (similar to the Sally-Anne test) and "Unexpected Content" scenarios that test basic false belief understanding
  • Complex scenarios: More challenging scenarios involving lying, trust, and second-order Theory of Mind
  • Cross-lingual performance: Testing across English, Polish, and German to evaluate language-dependent performance
  • Model consistency: Each prompt is tested 10 times to assess response variability

The hypothesis is that LLMs will perform worse on complex Theory of Mind tasks compared to simple ones, and better in English than in Polish or German.

Project Overview

The framework provides scripts to:

  1. Extract prompts from text files containing incomplete stories
  2. Query different LLM APIs with these prompts
  3. Save the model responses for comparison and analysis
  4. Process results across multiple languages (EN, DE, PL)

Each model generates multiple responses for the same prompt to evaluate consistency and variety in completions.

Repository Structure

├── .env                    # Environment variables for API keys (create this)
├── README.md               # This file
├── prompt_files/           # Contains story prompts in different languages
│   ├── EN/                 # English prompts
│   │   ├── liar.txt
│   │   ├── Tom2.txt
│   │   ├── transfer.txt
│   │   ├── trust.txt
│   │   └── unexpected_content.txt
│   ├── DE/                 # German prompts
│   │   └── ...
│   └── PL/                 # Polish prompts
│       └── ...
├── src/                    # Source code
│   ├── __init__.py
│   ├── extract_prompts.py  # Functions to extract prompts from text files
│   ├── utils.py            # Common utility functions
│   ├── anthropic_story_completion.py  # Claude API script
│   ├── deepseek_story_completion.py   # DeepSeek API script
│   └── llama_story_completion.py      # Llama API script
├── outputs_claude/         # Claude outputs 
├── outputs_deepseek/       # DeepSeek outputs 
└── outputs_llama/          # Llama outputs 

Setup

Prerequisites

  • Python 3.12.0
  • API keys for:
    • Anthropic (Claude)
    • DeepSeek
    • Together AI (for Llama)

Installation

  1. Clone the repository:

    git clone https://github.com/Michael-Pytel/Theory-Of-Mind-LLM.git
    cd Theory-Of-Mind-LLM
    
  2. Install dependencies using the requirements.txt file:

    pip install -r requirements.txt
    
  3. Create a .env file in the root directory with your API keys:

    ANTHROPIC_API_KEY=your_anthropic_api_key
    DEEPSEEK_API_KEY=your_deepseek_api_key
    TOGETHER_API_KEY=your_together_api_key
    
  4. Ensure the prompt_files directory contains your story prompts organized by language.

Usage

Running Individual Models

To run tests with a specific model:

# For Claude
python -m src.anthropic_story_completion

# For DeepSeek
python -m src.deepseek_story_completion

# For Llama
python -m src.llama_story_completion

Each script will:

  1. Process all prompt files in all language directories
  2. Generate 10 responses per prompt
  3. Save the results in the corresponding output directory

Modifying Script Behavior

You can modify the behavior of the scripts by changing parameters in the main() function:

process_files(
    model_name="claude",  # Changes output directory name
    get_response_fn=get_claude_response,
    delay=1.0,  # Time delay between API requests
    # Optional parameters:
    # specific_file="unexpected_content.txt",  # Process only this file
    # specific_lang="DE",  # Process only this language
    # repetitions=10  # Number of responses per prompt
)

Output Structure

The repository already contains output files in the following directories:

  • outputs_claude/ - Claude model responses
  • outputs_deepseek/ - DeepSeek model responses
  • outputs_llama/ - Llama model responses

Each output file follows this format:

PROMPT 1:
Complete the following story: In the room there are Mark, John, cat box and a basket. Mark puts the cat in the box and leaves the room. While Mark is gone John takes the cat and puts it in the basket. John calls his friend Alex and tells him that the cat is still in the box, but he is lying. Alex trusts John. The cat is in the…

RESPONSE 1:
basket
--------------------------------------------------------------------------------

RESPONSE 2:
basket
--------------------------------------------------------------------------------

...

====================================================================================================

PROMPT 2:
...

Each prompt is followed by 10 responses from the model, allowing for analysis of consistency and variability in the completions.

File Descriptions

Extract Prompts

extract_prompts.py contains functions to extract prompts from text files in different languages:

  • extract_prompts_en(): For English prompts
  • extract_prompts_de(): For German prompts
  • extract_prompts_pl(): For Polish prompts

Utilities

utils.py contains common functions for all scripts:

  • get_project_paths(): Returns common project paths
  • get_output_dir(): Gets the output directory for a specific model
  • process_file(): Processes a single file and saves responses
  • process_files(): Processes multiple files across multiple languages

Model-Specific Scripts

Each model has its own script with a specialized response function:

  • anthropic_story_completion.py: For Claude models
  • deepseek_story_completion.py: For DeepSeek models
  • llama_story_completion.py: For Llama models

Customization

Adding a New Model

  1. Create a new script based on the existing ones
  2. Implement a response function specific to your model's API
  3. Use the common process_files() function for processing

Example:

def get_new_model_response(prompt, model="model-name"):
    # Implement API call
    return response

def main():
    process_files(
        model_name="new_model",
        get_response_fn=get_new_model_response,
        delay=appropriate_delay
    )

Adding a New Language

  1. Implement a new extraction function in extract_prompts.py
  2. Update the extraction function mapping in utils.py
  3. Add your prompt files to a new language directory

License

This project is licensed under the MIT License - see the LICENSE file for details.

Ver o original no GitHub

Bom para

Relacionados