convez.ai
MCP ServerAutomaçãoAvançado
mcp-tool-shop-org

AI Jam Sessions

mcp-tool-shop-org/ai-jam-sessions

Ensine IA a tocar música com senses integradas: leitura, audição, visão e memória

Ver no GitHub
BomPluginScore · de 0 a 100

Servidor MCP que ensina IA a tocar piano, guitarra e cantar através de 120 músicas, 6 motores de áudio e um cockpit interativo no navegador. Inclui dataset público jam-actions-v0 com 115 registros de traces de uso de ferramentas MCP em piano clássico, com gate de liberação em 7 eixos.

Instalação

$npx convez-ai mcp-tool-shop-org/ai-jam-sessions

1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.

Métricas

Estrelas
5
Forks
1
Acompanhando
0
Saúde
100%
Último commit
há 1 semana
Versão
v2.9.0
Issues abertas
6
Contribuidores
2

Casos de uso

  • Treinar modelos de linguagem para compreender e gerar música
  • Pesquisa em aprendizado de máquina aplicado a artes criativas
  • Geração de datasets anotados para análise musical com IA

Veredito

Um projeto experimental sofisticado que combina síntese de áudio, anotação musical e treinamento de IA. Melhor aproveitado por pesquisadores e desenvolvedores que exploram criatividade musical com modelos de linguagem, não para casos de uso produtivos imediatos.

README

A tradução para o português deste README ainda está na fila.


What is this?

A piano and guitar that AI learns to play. Not a synthesizer, not a MIDI library — a teaching instrument.

An LLM can read and write text, but it can't experience music the way we do. No ears, no fingers, no muscle memory. AI Jam Sessions closes that gap by giving the model senses it can actually use:

  • Reading — real MIDI sheet music with deep musical annotations. Not hand-written approximations — parsed, analyzed, and explained.
  • Hearing — six audio engines (oscillator piano, sample piano, vocal samples, physical vocal tract, additive vocal synth, physically-modeled guitar) that play through your speakers, so the humans in the room become the AI's ears.
  • Seeing — a piano roll that renders what was played as SVG the model can read back and verify. An interactive guitar tablature editor. A browser cockpit with a visual keyboard, dual-mode note editor, and tuning lab.
  • Remembering — a practice journal that persists across sessions, so learning compounds over time.
  • Singing — vocal tract synthesis with 20 voice presets, from operatic soprano to electronic choir. Sing-along mode with solfege, contour, and syllable narration.

Each of the 12 genres has a richly annotated exemplar — a reference piece the AI studies first, with historical context, bar-by-bar structural analysis, key moments, teaching goals, and performance tips. The other 96 songs are raw MIDI, waiting for the AI to absorb the patterns, play the music, and write its own annotations.

Out of this same work, we also publish jam-actions-v0 — a public dataset of 115 multi-turn MCP tool-use traces over real classical piano. It teaches LLMs to do grounded tool-use over symbolic music, not just text generation, and ships with a 7-axis release gate that distinguishes "passing on evidence" from "passing because the task is trivial." See Training Dataset below for the full story.

The Piano Roll

The piano roll is how the AI sees music. It renders any song as SVG — blue for right hand, coral for left, with beat grids, dynamics, and measure boundaries:

Two color modes: hand (blue/coral) or pitch-class (chromatic rainbow — every C is red, every F# is cyan). The SVG format means the model can both see the image and read the markup to verify pitch, rhythm, and hand independence.

The Cockpit

A browser-based instrument and vocal studio that opens alongside the MCP server. No plugins, no DAW — just a web page with a piano.

  • Dual-mode piano roll — switch between Instrument mode (chromatic pitch-class colors) and Vocal mode (notes colored by vowel shape: /a/ /e/ /i/ /o/ /u/)
  • Visual keyboard — two octaves from C4, mapped to your QWERTY keyboard. Click or type.
  • 20 voice presets — 15 Kokoro-mapped voices (Aoede, Heart, Jessica, Sky, Eric, Fenrir, Liam, Onyx, Alice, Emma, Isabella, George, Lewis, plus choir and synth-vox), 4 tract-mapped voices, and a synthetic choir section
  • 10 instrument presets — the 6 server-side piano voices plus synth-pad, organ, bell, and strings
  • Note inspector — click any note to edit velocity, vowel, and breathiness
  • 7 tuning systems — Equal temperament, just intonation (major/minor), Pythagorean, quarter-comma meantone, Werckmeister III, or custom cent offsets. Adjustable A4 reference (392–494 Hz).
  • Tuning audit — frequency table, interval tester with beat-frequency analysis, and tuning export/import
  • Score import/export — serialize the entire score as JSON and load it back
  • LLM-facing API — window.__cockpit exposes exportScore(), importScore(), addNote(), play(), stop(), panic(), setMode(), and getScore() so an LLM can compose, arrange, and play back programmatically

The Learning Loop

 Read                 Play                See                 Reflect
┌──────────┐     ┌───────────┐     ┌────────────┐     ┌──────────────┐
│ Study the │     │ Play the  │     │ View the   │     │ Write what   │
│ exemplar  │ ──▶ │ song at   │ ──▶ │ piano roll │ ──▶ │ you learned  │
│ analysis  │     │ any speed │     │ to verify  │     │ in journal   │
└──────────┘     └───────────┘     └────────────┘     └──────┬───────┘
                                                             │
                                                             ▼
                                                    ┌──────────────┐
                                                    │ Next session  │
                                                    │ picks up here │
                                                    └──────────────┘

The Song Library

120 songs across 12 genres, built from real MIDI files. Each genre has one deeply annotated exemplar — with historical context, bar-by-bar harmonic analysis, key moments, teaching goals, and performance tips (including vocal guidance). These exemplars serve as templates: the AI studies one, then annotates the rest.

GenreExemplarKeyWhat it teaches
BluesThe Thrill Is Gone (B.B. King)B minorMinor blues form, call-and-response, playing behind the beat
ClassicalFür Elise (Beethoven)A minorRondo form, touch differentiation, pedaling discipline
FilmComptine d'un autre été (Tiersen)E minorArpeggiated textures, dynamic architecture without harmonic change
FolkGreensleevesE minor3/4 waltz feel, modal mixture, Renaissance vocal style
JazzAutumn Leaves (Kosma)G minorii-V-I progressions, guide tones, swing eighths, rootless voicings
LatinThe Girl from Ipanema (Jobim)F majorBossa nova rhythm, chromatic modulation, vocal restraint
New-AgeRiver Flows in You (Yiruma)A majorI-V-vi-IV recognition, flowing arpeggios, rubato
PopImagine (Lennon)C majorArpeggiated accompaniment, restraint, vocal sincerity
RagtimeThe Entertainer (Joplin)C majorOom-pah bass, syncopation, multi-strain form, tempo discipline
R&BSuperstition (Stevie Wonder)Eb minor16th-note funk, percussive keyboard, ghost notes
RockYour Song (Elton John)Eb majorPiano ballad voice-leading, inversions, conversational singing
SoulLean on Me (Bill Withers)C majorDiatonic melody, gospel accompaniment, call-and-response

Songs progress from raw (MIDI only) → annotated → ready (fully playable with musical language). The AI promotes songs by studying them and writing annotations with annotate_song.

Sound Engines

Six engines, plus a layered combinator that runs any two simultaneously:

EngineTypeWhat it sounds like
Oscillator PianoAdditive synthesisMulti-harmonic piano with hammer noise, inharmonicity, 48-voice polyphony, stereo imaging. Zero dependencies.
Sample PianoWAV playbackSalamander Grand Piano — 480 samples, 16 velocity layers, 88 keys. The real thing.
Vocal (Sample)Pitch-shifted samplesSustained vowel tones with portamento and legato mode.
Vocal TractPhysical modelPink Trombone — LF glottal waveform through a 44-cell digital waveguide. Four presets: soprano, alto, tenor, bass.
Vocal SynthAdditive synthesis15 Kokoro voice presets with formant shaping, breathiness, vibrato. Deterministic (seeded RNG).
GuitarAdditive synthesisPhysically-modeled plucked string — 4 presets (steel dreadnought, nylon classical, jazz archtop, twelve-string), 8 tunings, 17 tunable parameters.
LayeredCombinatorWraps two engines and dispatches every MIDI event to both — piano+synth, vocal+synth, etc.

Keyboard Voices

Six tunable piano voices, each adjustable per-parameter (brightness, decay, hammer hardness, detune, stereo width, and more):

VoiceCharacter
Concert GrandRich, full, classical
UprightWarm, intimate, folk
Electric PianoSilky, jazzy, Fender Rhodes feel
Honky-TonkDetuned, ragtime, saloon
Music BoxCrystalline, ethereal
Bright GrandCutting, contemporary, pop

Guitar Voices

Four guitar voice presets with physically-modeled string synthesis, each with 17 tunable parameters (brightness, body resonance, pluck position, string damping, and more):

VoiceCharacter
Steel DreadnoughtBright, balanced, classic acoustic
Nylon ClassicalWarm, soft, rounded
Jazz ArchtopMellow, woody, clean
Twelve-StringShimmering, doubled, chorus-like

The Practice Journal

After every session, the server captures what happened — which song, what speed, how many measures, how long. The AI adds its own reflections: what it noticed, what patterns it recognized, what to try next.

---
### 14:32 — Autumn Leaves
**jazz** | intermediate | G minor | 69 BPM × 0.7 | 32/32 measures | 45s

The ii-V-I in bars 5-8 (Cm7-F7-BbMaj7) is the same gravity as the V-i
in The Thrill Is Gone, just in major. Blues and jazz share more than the
genre labels suggest.

Next: try at full speed. Compare the Ipanema bridge modulation with this.
---

One markdown file per day, stored in ~/.ai-jam-sessions/journal/. Human-readable, append-only. Next session, the AI reads its journal and picks up where it left off.

Training Dataset

jam-actions-v0 — a public dataset of multi-turn MCP tool-use traces grounded in real classical-piano MIDI. Built from the same library this server teaches with, the dataset teaches LLMs to do grounded tool-use over symbolic music — not just text generation.

Each record pairs a 4-measure phrase window with an annotated teaching target and a target trace — a turn-by-turn session in which an assistant uses the MCP tools above (get_events_in_measure, get_events_in_hand, count_distinct_pitch_classes, and the rest of the 9-tool MIDI inspector surface) to read, analyze, and discuss the phrase.

DOI10.5281/zenodo.20279919 — Zenodo, published 2026-05-19
Records115 (public subset)
Canonical baseline16-record post-repair E3
Compositions8 classical piano works (Beethoven, Bach, Schubert, Schumann, Mozart, Mendelssohn, Tchaikovsky)
Source MIDIpiano-midi.de — Bernd Krueger arrangements
LicenseCC-BY-SA-3.0-DE (arrangements) over public-domain compositions
Version0.4.3 (2026-05-19)
Schemarelease-gate-assessment/2.0.0

Quality story — the 7-axis release gate. The dataset ships with a release gate that distinguishes evidence-grounded passing from ceiling-saturated passing. Axes 1–6 are blocking (absolute floor, margin compound, tool-use rate, correct-after-tool, misinterpretation count, stratum floor); axis 7 is enriched-vs-non reporting. Axes 2 and 6 admit a ceiling_saturated_pass bucket so records that score 1.000 across text-only / tool-inspected / random-MIDI conditions don't dilute the harder strata. The Slice 22 baseline PASSES the revised gate. The Slice 19 baseline still FAILS it — kept as a regression diagnostic so the gate has teeth.

Reproducibility. A fresh contributor on any platform (Windows native, macOS, Linux, WSL) can verify the package and reproduce the canonical PASS verdict in under a minute:

git clone https://github.com/mcp-tool-shop-org/ai-jam-sessions.git
cd ai-jam-sessions && pnpm install
pnpm exec tsx scripts/verify-public-package-checksums.ts        # 273 entries, ~2s
pnpm exec tsx scripts/check-release-gate.ts \
  datasets/jam-actions-v0-public/evals/slice21-fair-e3-baseline-results.json
# → "Verdict: PASS"

.gitattributes pins LF line endings for *.sha256 and the public-dataset tree so the checksum verifier works on every platform. The release-gate CLI is strict-positional (rejects unknown / multiple positional args) so cold-start contributors can't silently mis-invoke it.

Where to find it. The published Zenodo record is at https://zenodo.org/records/20279919 (DOI: 10.5281/zenodo.20279919). The full dataset card is at datasets/jam-actions-v0-public/README.md. Zenodo deposition metadata is at zenodo-metadata.json, citation metadata at CITATION.cff, the publication receipt at publication-receipt.json, and release notes at RELEASE_NOTES.md. The 25-slice build arc — from initial corpus draft through the off-by-one repair, the Schumann remediation, the RC-gate revision, the operator-aloneness audit, and the publication execution — lives in docs/.

Cite it. mcp-tool-shop-org & Krueger, B. (2026). AI Jam Sessions — Tool-Use Traces v0 (Public Subset). Zenodo. https://doi.org/10.5281/zenodo.20279919

HuggingFace mirror. Coming in a v1.4.x patch — see datasets/jam-actions-v0-public/publication-receipt.json for the deferred-status block. The Zenodo DOI is the canonical citation handle; the HF mirror is for ML-ecosystem discovery only.

The MIDI arrangements are by Bernd Krueger (piano-midi.de), licensed CC-BY-SA-3.0-DE. The annotations, traces, and eval artifacts are by the AI Jam Sessions team, released under the same license so the share-alike chain is preserved end-to-end.

Install

npm install -g @mcptoolshop/ai-jam-sessions

Requires Node.js 18+. No MIDI drivers, no virtual ports, no external software.

Claude Desktop / Claude Code

{
  "mcpServers": {
    "ai_jam_sessions": {
      "command": "npx",
      "args": ["-y", "-p", "@mcptoolshop/ai-jam-sessions", "ai-jam-sessions-mcp"]
    }
  }
}

MCP Tools

41 tools and 3 prompt templates across six categories:

Learn

ToolWhat it does
list_songsBrowse by genre, difficulty, or keyword
song_infoFull musical analysis — structure, key moments, teaching goals, style tips
registry_statsLibrary-wide stats: total songs, genres, difficulties
list_measuresEvery measure's notes, dynamics, and teaching notes
teaching_noteDeep dive into a single measure — fingering, dynamics, context
suggest_songRecommendation based on genre, difficulty, and what you've played
practice_setupRecommended speed, mode, voice settings, and CLI command for a song
compare_songsCross-genre pattern recognition — key relationships, pitch/interval similarity, shared forms, teaching connections
annotation_progressTrack annotation quality across the library — scores, grades, and improvement suggestions
server_infoServer version, library stats, engine list, active session

Play

ToolWhat it does
play_songPlay through speakers — library songs or raw .mid files. Any engine, speed, mode, measure range.
stop_playbackStop
pause_playbackPause or resume
set_speedChange speed mid-playback (0.1×–4.0×)
playback_statusReal-time snapshot: current measure, tempo, speed, keyboard voice, state
view_piano_rollRender as SVG (hand color or pitch-class chromatic rainbow)
score_performanceScore a MIDI play-along — pitch accuracy, timing, completeness, with graded feedback
mute_handMute or unmute left/right hand during practice — isolate one hand at a time
preview_teaching_cuesSee all teaching notes and key moments before playing

Sing

ToolWhat it does
sing_alongSingable text — note-names, solfege, contour, or syllables. With or without piano accompaniment.
ai_jam_sessionsGenerate a jam brief — chord progression, melody outline, and style hints for reinterpretation

Guitar

ToolWhat it does
view_guitar_tabRender interactive guitar tablature as HTML — click-to-edit, playback cursor, keyboard shortcuts
list_guitar_voicesAvailable guitar voice presets
list_guitar_tuningsAvailable guitar tuning systems (standard, drop-D, open G, DADGAD, etc.)
tune_guitarAdjust any parameter of any guitar voice. Persists across sessions.
get_guitar_configCurrent guitar voice config vs factory defaults
reset_guitarFactory reset a guitar voice

Build

ToolWhat it does
add_songAdd a new song as JSON
import_midiImport a .mid file with metadata
annotate_songWrite musical language for a raw song and promote it to ready
save_practice_noteJournal entry with auto-captured session data
read_practice_journalLoad recent entries for context
list_keyboardsAvailable keyboard voices
tune_keyboardAdjust any parameter of any keyboard voice. Persists across sessions.
get_keyboard_configCurrent config vs factory defaults
reset_keyboardFactory reset a keyboard voice
score_annotationScore annotation quality across 5 dimensions — completeness, depth, specificity, teaching value, vocabulary
validate_song_entryValidate a song JSON against the schema before adding
transpose_songTranspose a song up or down by semitones — new key, new notes
list_sectionsView structural sections of a song (Intro, Verse, Chorus, etc.)
add_sectionAdd a section marker to a song for structural navigation

MCP Prompts

Three prompt templates for structured teaching workflows:

PromptWhat it does
annotate_songGuided annotation workflow — study an exemplar, write musical language for a raw song
practice_planBuild a structured practice plan based on genre, difficulty, and goals
performance_reviewReview a completed session — what went well, what to focus on next

CLI

ai-jam-sessions list [--genre <genre>] [--difficulty <level>]
ai-jam-sessions play <song-id> [--speed <mult>] [--mode <mode>] [--engine <piano|vocal|tract|synth|guitar|piano+synth|guitar+synth>]
ai-jam-sessions sing <song-id> [--with-piano] [--engine <engine>]
ai-jam-sessions view <song-id> [--measures <start-end>] [--out <file.svg>]
ai-jam-sessions view-guitar <song-id> [--measures <start-end>] [--tuning <tuning>]
ai-jam-sessions info <song-id>
ai-jam-sessions tune <keyboard-id> [--param value ...] [--reset] [--show]
ai-jam-sessions tune-guitar <voice-id> [--param value ...] [--reset] [--show]
ai-jam-sessions keyboards
ai-jam-sessions guitars
ai-jam-sessions stats
ai-jam-sessions library
ai-jam-sessions ports
ai-jam-sessions help
ai-jam-sessions --version

Status

v1.4.1. Six sound engines, 41 MCP tools, 3 prompt templates, 120 songs across 12 genres with deeply annotated exemplars. Song transposition, section markers, per-hand mute/solo for focused practice. Interactive guitar tablature editor. Browser cockpit with 20 vocal presets, 10 instrument voices, 7 tuning systems, and an LLM-facing score API. Piano roll visualization in two color modes. Practice journal for persistent learning. Session state persistence across server restarts. MIDI play-along scoring, annotation quality assessment, and cross-genre pattern recognition.

Also publishes jam-actions-v0 — a 115-record training dataset of multi-turn MCP tool-use traces over classical piano, with a 7-axis release gate, cold-start reproducibility, and full Zenodo + CITATION.cff metadata (CC-BY-SA-3.0-DE). 1513 tests passing across the MCP server + dataset packagers + eval harnesses + release-gate validator. The MIDI is all there — the library grows as the AI learns, and now there's a corpus of that learning shipped with it.

Security & Privacy

Data touched: song library (JSON + MIDI), user songs directory (~/.ai-jam-sessions/songs/), guitar tuning configs, practice journal entries, local audio output device.

Data NOT touched: no cloud APIs, no user credentials, no browsing data, no system files outside the user song directory. No telemetry is collected or sent.

Permissions: MCP server uses stdio transport only (no HTTP). CLI accesses local filesystem and audio devices. See SECURITY.md for the full policy.

License

MIT

Ver o original no GitHub

Bom para

Relacionados