Mirelo and Kyutai Open-Source MuScriptor, an Audio-to-MIDI Model That Transcribes Full Mixes

Mirelo AI and Kyutai open-sourced MuScriptor, an audio-to-MIDI model that takes a finished music mix and transcribes every instrument at once into separate MIDI tracks. Most audio-to-MIDI tools handle one sound at a time. MuScriptor reads the whole mix.

The model outputs a separate MIDI track per instrument on a piano roll, with automatic instrument detection and labeling. It also detects chords, key, and tempo, and supports one-click export. The weights are on Hugging Face, the inference code is on GitHub, and a companion tool ships free inside Mirelo Studio.

Transcription treated as a language modeling problem

MuScriptor uses a decoder-only transformer that accepts mel-spectrograms and autoregressively generates token sequences representing pitch, timing, and instrument type. Mirelo frames the approach as "music transcription as language modeling," borrowing the same architecture pattern that powers text generation and applying it to notes instead of words.

The design targets a specific failure point. Dense, multi-instrument audio has historically tripped up transcription models built to isolate a single voice or instrument. According to Mirelo, MuScriptor "takes the full mix and transcribes every instrument at once" rather than requiring source separation as a first step.

A three-stage training pipeline mixing synthetic and real audio

The model was trained across three datasets, each doing a different job:

• DSynth. Roughly 1.45 million MIDI files rendered to audio with on-the-fly synthesis, giving the model broad coverage of note and instrument combinations.

• DReal. About 170,000 real recordings, more than 11,000 hours, paired with aligned annotations so the model learns how studio audio actually sounds.

• DRL. A smaller set of 300 high-quality tracks used for a GRPO-like reinforcement learning stage to sharpen output quality.

Fine-tuning on real recordings mattered. Mirelo reports that training on real data improved transcription metrics by roughly 20 percentage points over a version pre-trained on synthetic audio alone, per the team's ArXiv paper, which credits IRCAM in its citation.

Why per-instrument MIDI matters for scoring and remixing

For composers, editors, and music supervisors, MIDI is editable in a way that audio is not. A transcription that separates a mix into per-instrument tracks turns a reference recording into a starting point: swap a synth patch, re-voice a chord, quantize a drum pattern, or lift a bassline into a new arrangement. Getting there previously meant either manual transcription or stems that still had to be transcribed one instrument at a time.

MuScriptor lands amid a run of open and accessible music AI. We covered Google's Lyria 3, which generates full tracks with vocals from text prompts.

We also covered ElevenLabs Music Finetunes, which trains a custom music model on a user's own catalog. MuScriptor works the opposite direction: instead of generating audio, it reads existing audio back into a symbolic, editable format.

Open weights lower the barrier for building on top

Because MuScriptor ships with open weights and inference code, developers can run it locally or wire it into their own tools rather than depending on a hosted API. The decoder-only, language-modeling framing also means the architecture will feel familiar to teams already working with transformer models.

For anyone who wants to try it without touching code, the audio-to-MIDI tool in Mirelo Studio is free and runs the improved model. The combination of a free hosted tool and open weights gives both casual users and builders a way in.