节奏舞者:基于关键动作转换图和有条件姿态插值网络的3D舞蹈生成方法研究

Rhythm dancer:3D dance generation by key-motion transition graph and pose-interpolation network

3D舞蹈是元宇宙中虚拟人的一种重要表现形式,它将音乐与舞蹈进行有机结合,大大增强了元宇宙中相关应用的趣味性。之前的工作通常把3D舞蹈生成简单视作一个序列生成任务,但是生成的舞蹈动作质量较差且与音乐的契合度较低。受人类学习舞蹈过程的启发,提出了一种新颖的3D舞蹈框架——“节奏舞者”来解决上述问题。该框架首先使用VQ-VAE-2对舞蹈进行分层编码量化,可有效改善舞蹈生成质量;然后使用节奏点上的关键动作编码建立关键动作转换图,既可保证生成的舞蹈动作与音乐节拍的契合度,又可增加舞蹈动作的多样性。为了确保关键动作之间平滑自然地连接,提出了一个姿态插值网络来学习关键动作之间的转换动作。通过大量实验证明,该框架避免了长序列生成的不稳定和不可控问题,实现了舞蹈动作与音乐节奏的高度契合,达到了当前最优效果。

3D dance is an indispensable form of virtual humans in the metaverse.It organically combines music and dance art,which greatly increases the interest in the metaverse.Previous work usually treats it as a simple sequence generation task,but it is difficult to match the dance movements with the music beat perfectly and the quality of long sequence dance generation is difficult to be guaranteed.Inspired by the process by which humans learn to dance,a novel 3D dance framework“Rhythm Dancer”to solve the above problems was proposed.The framework first uses VQ-VAE-2 to encode and quantify the dances in a hierarchical way,which effectively improves the quality of dance generation.Then,a key movement transition map was created using the core dance movements on the rhythm points,which not only ensures that the generated dance movements fit with the music beat,but also increases the diversity of dance movements.To ensure smooth and natural connections between the core dance moves,a pose-interpolation network was proposed to learn the transition movements between key moves.Extensive experiments demonstrate that the framework not only avoids the instability and uncontrollability problems of long sequence generation,but also achieves a higher match between dance movements and music rhythms,reaching state-of-the-art results.