Con Moto

December / 2025 / Embodied AI duet performance

Published at NIME 2026
Performed in Paine Hall, Harvard

Embodied steering of music transformers for live dance improvisation

Con Moto system diagram: live dance, depth camera, pose estimation, motion analysis, Max/MSP control hub, real-time music transformer, MIDI rendering, mixer and sound output

A dancer's movement steers an AI music model in real time: where the dancer stands and how they move sets the range and instruments it plays.

I worked with Heidi Lei, a generative music researcher, for three months. She handled the model, and I built the camera tracking, the sound design, the movement to music mapping, and the live rig across three computers and eight speakers. Halfway through we brought in a dancer trained in ballet and Hip-Hop as a co-designer, and she and I performed it as a duet for a live audience. It was published at NIME 2026 in London.

System

A depth camera tracks the dancers, and Max/MSP turns their position, activity and movement motifs into control messages. The music model keeps generating, and when a message arrives it re-steers: notes outside the active pitch range or the enabled instruments are masked out before it samples, so the dancer shapes the register while the model still chooses the notes and rhythms. In parallel, movement drives timbre and dynamics in Ableton Live.

The system runs across three devices: a Windows laptop for tracking, a MacBook running Max/MSP and Ableton Live, and a PC running the music generation engine, with eight-channel audio sent to the mixer over Dante.

The full Con Moto system diagram, from live dance through the camera, pose estimation, motion analysis, Max/MSP, the music transformer and MIDI rendering to the mixer and sound output — The full system diagram. Tap to open it full size.

Diagram: the model's pitch probabilities, a mask set by the dancer's position, and the masked distribution between A3 and C5 over a piano keyboard — The pitch-token probabilities from the unconstrained model are masked to enforce a stage-position-dependent pitch range.

unscored

unscored is an improvised performance where the musical structure is entirely emergent. Abena Kyereme-Tuah and I performed it as my final project for Professor Hans Tutschku's class, Music 264, Composing with Max/MSP, at the Harvard Hydra concert in Paine Hall on December 12, 2025. It is in two acts, each with two scenes.

Discovery. A four-minute solo on a model fine-tuned on classical music: floor position sets pitch and spatialization, and joint velocities set how loud the notes are.
Connection. The second dancer joins with a second voice, and the distance between us adds distortion.
Exchange. A collapse switches to a model fine-tuned on jazz. We each embody an instrument that plays only when we step into the spotlight.
Battle. The music fades to silence, the system restarts on its own, and we return for a battle of Jazz against Disco. The Battle's tracks were generated offline for stability.

“We were not just dancing with a machine; we were dancing with each other through the machine.”

Abena Kyereme-Tuah and Zhixing Chen dancing unscored on stage in Paine Hall

Zhixing Chen dancing the opening solo on the Paine Hall stage, audience in the foreground — Discovery.

The two dancers face to face at centre stage — Connection.

Zhixing Chen low to the floor during the Battle — Battle.

Abena Kyereme-Tuah dancing during the Battle — Battle.

Team

Heidi Lei (MIT MEng 2026) on the generative model, and Abena Kyereme-Tuah, a professional dancer trained in ballet and Hip-Hop, as co-designer and performer.

Zhixing Chen, Heidi Lei and Cheng-Zhi Anna Huang, Con Moto: Embodied Steering of Music Transformers for Live Dance Improvisation, Proceedings of the International Conference on New Interfaces for Musical Expression (NIME '26), London, June 2026, pp. 50–59.

Zhixing Chen[email protected]