multimodal-music-intelligence-system
SriChandhana/multimodal-music-intelligence-system
Agentic AI system with custom MCP server orchestrating 7 AI tools across Spotify, MusicBrainz & ACRCloud — multimodal music intelligence (audio, text, image)
Instalação
npx convez-ai SriChandhana/multimodal-music-intelligence-system1 comando no terminal: detecta o formato do repositório (marketplace ou plugin avulso) e instala via Claude Code. Requer Node 18+ e o Claude Code instalado.
Métricas
README
A tradução para o português deste README ainda está na fila.
🎵 MMMD — Multimodal Music Intelligence System
An AI-powered multimodal music intelligence platform that identifies, analyzes, and enriches music information from audio, text, and image inputs. The system uses a custom MCP (Model Context Protocol) server to orchestrate multiple tools and external APIs, enabling intelligent music discovery, metadata enrichment, Indian classical music analysis, and contextual insights.
✨ Features
🎵 Audio-Based Music Recognition
- Song identification using ACRCloud audio fingerprinting
- Metadata extraction and enrichment
- Recognition confidence scoring
🔍 Text-Based Music Intelligence
- Natural language music queries
- Artist-based search with curated song recommendations
- Song metadata retrieval and enrichment
🖼️ Image-Based Music Recognition
- Claude Vision integration for music-related image understanding
- Multimodal analysis pipeline
🎼 Indian Classical Music Analysis
- Raaga detection and enrichment
- Taala and rasa analysis
- Composer and music system identification
- Carnatic music knowledge retrieval
🤖 Music Intelligence
- AI-generated song summaries
- Similar song recommendations
- Contextual song insights
- Metadata aggregation from multiple sources
🏗️ Architecture
Input Layer
[Audio] [Text] [Image]
│
▼
┌─────────────────┐
│ Custom MCP │
│ Server │
│ (7 AI Tools) │
└────────┬────────┘
│
▼
Tool Router
┌─────────┼─────────┐
│ │ │
▼ ▼ ▼
ACRCloud Spotify MusicBrainz
(Audio ID) (Meta) (Credits)
│ │ │
└─────────┼─────────┘
▼
Metadata Aggregation
│
▼
┌───────────────────────┐
│ Music Intelligence │
│ Layer (Claude API + │
│ LLM Reasoning) │
└───────────┬───────────┘
│
▼
Indian Classical Analysis
(Raaga Detection)
│
▼
Summary & Recommendations
│
▼
Unified Response
🛠️ Tech Stack
Frontend
- React
- TypeScript
- Vite
- Tailwind CSS
Backend
- Node.js
- Express.js
- TypeScript
AI & Orchestration
- Claude API
- MCP (Model Context Protocol)
- ACRCloud
External APIs
- Spotify API
- MusicBrainz API
🚀 Key Capabilities
- Multimodal music understanding through audio, text, and image inputs
- Custom MCP orchestration layer coordinating multiple intelligence tools
- Indian classical music enrichment beyond traditional music metadata systems
- Unified analysis pipeline combining external APIs with domain-specific knowledge bases
- Full-stack TypeScript architecture with modular service design
📸 Screenshots
Home Page
Audio Recognition
Text Search
Indian Music Analysis
Similar Song Recommendations
Artist Lookup
AI Summary
⚙️ Setup
Clone Repository
git clone https://github.com/SriChandhana/multimodal-music-intelligence-system.git
Backend
cd backend
npm install
npm run dev
Frontend
cd frontend
npm install
npm run dev
Environment Variables
Create:
PORT=5000
ACR_HOST=
ACR_ACCESS_KEY=
ACR_ACCESS_SECRET=
SPOTIFY_CLIENT_ID=
SPOTIFY_CLIENT_SECRET=
🔮 Future Enhancements
- Enhanced recommendation engine
- Expanded Indian classical music knowledge base
- Artist profile enrichment
- Advanced semantic music search
- Production deployment
👩💻 Author
Kandula Sri Chandhana
B.Tech CSE (AI & ML), VNR VJIET