WHAT THE RESEARCH SHOWS

Music in,
movement out.

Before a computer can dance, real dancers have to.

Music-to-dance models learn from thousands of recorded performances turned into 3D motion. Here is where that data comes from, what the models can do and where they still slip.

5.2 h of 3D dance motion

Check an edit

A street dancer mid-move in a bright studio, watched by several video cameras on tripods placed around the floor — AI-generated image, not a photograph of a real person or place.

Dancers first
Beat and body together
Watch the feet
Dance research, in plain steps

THE SHORT ANSWER

The data is human,
the moves are new.

Models learn from recorded dancers,
then generate motion for music they have not heard.

THE DATA

Recorded from real dancers

One key collection, the AIST Dance Video Database, holds 13,939 street dance videos by 40 professional dancers in 10 genres, with original, copyright-cleared music.

See the numbers

THE MODELS

Music in, motion out

Researchers turned those videos into 3D motion and trained models to generate dance from music. A plain sequence model was not enough: its moves did not match the music well.

Check an edit

THE LIMITS

Feet that slide

Generated dancers can glide when they should grip. The EDGE team trained its model to tell intentional slides from unintentional ones, and added a new measure of physical plausibility.

What to look for

A QUICK EDIT CHECK

How much of the dance
do you give it?

Editable dance models can start from different amounts of human input. Pick one to see what the EDGE model does with it. The bar shows how much of the dance you supply.

Edit check

Choose what you supplyJust the musicAn opening moveThe first and last movesHalf the body

Just the musicFull generation

The model encodes the music with a separate music model, Jukebox, then generates the whole dance. EDGE is trained on 5-second clips but stitches them into dances of any length.

An opening moveContinuation

You give the first movements and the model carries on from there, keeping the motion consistent with what came before.

The first and last movesIn-betweening

You fix how the dance starts and ends, and the model fills in the movement between them.

Half the bodyJoint-wise editing

You supply the upper body’s movement and the model generates the lower body to match, or the other way round.

Source: EDGE: Editable Dance Generation from Music, project page and paper abstract (Tseng, Castellon and Liu, Stanford University).

Based on the EDGE project (Stanford University, CVPR 2023). Research systems, not consumer products.

TWO NUMBERS WORTH KNOWING

Big datasets,
short clips.

One figure on the source videos,
one on the 3D motion built from them.

13,939

Original street dance videos

The AIST Dance Video Database covers break, pop, lock, waack, middle hip-hop, LA-style hip-hop, house, krump, street jazz and ballet jazz, filmed with up to nine cameras around the dancer.

SOURCE VIDEOS · AIST

5.2 hours

Hours of 3D dance motion with music

AIST++ turned the videos into 1,408 sequences of 3D dance motion, spread evenly across the 10 genres. Each lasts between 7.4 and 48.0 seconds, and every one has its music.

3D MOTION · AIST++

The database’s 60 music tracks give each genre six tempos, from 80 to 130 beats per minute, or 110 to 135 for house, since slow tempos do not suit house.

FOUR THINGS TO LOOK FOR

Watching a
generated dance.

Researchers judge generated dance on physical plausibility, beat alignment and variety, and by asking people which dances they prefer. You can look for the same things.

Watch the feet
Unintended foot sliding is a known flaw. Real dance has deliberate slides too, which is why EDGE learns when feet should and should not slide.
Listen for the beat
Good generated dance lines up with the music. Beat alignment is one of the measures researchers use to compare models.
Ask where the data came from
The AIST database was built from original videos with copyright-cleared music so that researchers could share it openly.
Treat it as a sketch
Editing tools such as in-betweening let a person fix key moments and let the model fill the gaps, rather than accept a whole dance as given.

A laptop with its screen turned away and a small speaker on the floor of an empty dance studio with a mirror wall — AI-generated image, not a photograph of a real person or place.

Close-up of a dancer's trainers turning on a pale wooden studio floor, caught with slight motion blur — AI-generated image, not a photograph of a real person or place.

FROM VIDEO TO 3D

Nine cameras,
millions of frames.

To build AIST++, researchers used the multi-camera videos to estimate camera positions, 3D body keypoints and full 3D motion. The result labels 10.1 million images of 30 dancers seen from nine views.

The original database splits its genres into Old School styles from about the 1970s to 1990s, such as break, pop, lock and waack, and New School styles since about the 1990s, such as house, krump and street jazz.

Questions

Questions people ask

Can an AI really make up a dance?

Research models can generate new 3D dance motion for music they were not trained on. They learn from recorded human performances, such as the 1,408 motion sequences in AIST++.

Why do generated dancers look odd?

One common giveaway is feet sliding across the floor when they should be planted. The EDGE researchers built a contact consistency loss to reduce unintended sliding while keeping deliberate slides.

Where does dance training data come from?

The AIST Dance Video Database was recorded with 40 professional dancers in a studio, using original music composed for 10 street dance genres and copyright cleared for research use.

How do researchers judge a generated dance?

EDGE was tested on physical plausibility, beat alignment and diversity, and in a large user study in which human raters strongly preferred its dances over two earlier methods.

Can these models help a choreographer?

EDGE was designed to be edited: it can continue a given opening, fill in between set start and end moves, or generate the lower body to match a given upper body.

Music in,
movement out.