Path guided motion synthesis for Drosophila larvae

The deformability and high degree of freedom of mollusks bring challenges in mathematical modeling and synthesis of motions.Traditional analytical and statistical models are limited by either rigid skeleton assumptions or model capacity,and have difficulty in generating realistic and multi-pattern mollusk motions.In this work,we present a large-scale dynamic pose dataset of Drosophila larvae and propose a motion synthesis model named Path2Pose to generate a pose sequence given the initial poses and the subsequent guiding path.The Path2Pose model is further used to synthesize long pose sequences of various motion patterns through a recursive generation method.Evaluation analysis results demonstrate that our novel model synthesizes highly realistic mollusk motions and achieves state-of-the-art performance.Our work proves high performance of deep neural networks for mollusk motion synthesis and the feasibility of long pose sequence synthesis based on the customized body shape and guiding path.

Music-stylized hierarchical dance synthesis with user control

Background Synthesizing dance motions to match musical inputs is a significant challenge in animation research.Compared to functional human motions,such as locomotion,dance motions are creative and artistic,often influenced by music,and can be independent body language expressions.Dance choreography requires motion content to follow a general dance genre,whereas dance performances under musical influence are infused with diverse impromptu motion styles.Considering the high expressiveness and variations in space and time,providing accessible and effective user control for tuning dance motion styles remains an open problem.Methods In this study,we present a hierarchical framework that decouples the dance synthesis task into independent modules.We use a high-level choreography module built as a Transformer-based sequence model to predict the long-term structure of a dance genre and a low-level realization module that implements dance stylization and synchronization to match the musical input or user preferences.This novel framework allows the individual modules to be trained separately.Because of the decoupling,dance composition can fully utilize existing high-quality dance datasets that do not have musical accompaniments,and the dance implementation can conveniently incorporate user controls and edit motions through a decoder network.Each module is replaceable at runtime,which adds flexibility to the synthesis of dance sequences.Results Synthesized results demonstrate that our framework generates high-quality diverse dance motions that are well adapted to varying musical conditions and user controls.

Introducing semantic information into motion graph

To improve motion graph based motion synthesis,semantic control was introduced.Hybrid motion features including both numerical and user-defined semantic relational features were extracted to encode the characteristic aspects contained in the character's poses of the given motion sequences.Motion templates were then automatically derived from the training motions for capturing the spatio-temporal characteristics of an entire given class of semantically related motions.The data streams of motion documents were automatically annotated with semantic motion class labels by matching their respective motion class templates.Finally,the semantic control was introduced into motion graph based human motion synthesis.Experiments of motion synthesis demonstrate the effectiveness of the approach which enables users higher level of semantically intuitive control and high quality in human motion synthesis from motion capture database.

Earthquake motion input and its dissemination via the Internet

Objectives of this task are to conduct research on seismic hazards,and to provide relevant input on the expected levels of these hazards to other tasks.Other tasks requiring this input include those dealing with inventory,fragility curves, rehabilitation strategies and demonstration projects.The corresponding input is provided in various formats depending on the intended use:as peak ground motion parameters and/or response spectral values for a given magnitude,epicentral distance and site conditions;or as time histories for scenario earthquakes that are selected based on the disaggregated seismic hazard mapped by the U.S.Geological Survey and are incorporated in building codes.The user community for this research is both academic researchers and practicing engineers who may use the seismic input generated by the synthesis techniques that are developed under this task for a variety of applications.These include ground motions for scenario earthquakes,for developing fragility curves and in specifying ground motion input for critical facilities (such as hospitals) located in the eastern U.S.

MULTITASK SCHEDULING IN NETWORKED CONTROL SYSTEMS WITH APPLICATION TO LARGE SCALE VEHICLE CONTROL

Aiming at scheduling problems of networked control system （NCS） used to fulfill motion synthesis and cooperation control of the distributed multi-mechatronic systems, the differences of network scheduling and task scheduling are compared, and the mathematic description of task scheduling is presented. A performance index function of task scheduling of NCS according to task balance and traffic load matching principles is defined. According to this index, a static scheduling method is designed and implemented to controlling task set simulation of the DCY100 transportation vehicle. The simulation results are applied successfully to practical engineering in this case so as to validate the effectiveness of the proposed performance index and scheduling algorithm.

A learning-based control pipeline for generic motor skills for quadruped robots

Performing diverse motor skills with a universal controller has been a longstanding challenge for legged robots.While motion imitation-based reinforcement learning(RL)has shown remarkable performance in reproducing designed motor skills,the trained controller is only suitable for one specific type of motion.Motion synthesis has been well developed to generate a variety of different motions for character animation,but those motions only contain kinematic information and cannot be used for control.In this study,we introduce a control pipeline combining motion synthesis and motion imitation-based RL for generic motor skills.We design an animation state machine to synthesize motion from various sources and feed the generated kinematic reference trajectory to the RL controller as part of the input.With the proposed method,we show that a single policy is able to learn various motor skills simultaneously.Further,we notice the ability of the policy to uncover the correlations lurking behind the reference motions to improve control performance.We analyze this ability based on the predictability of the reference trajectory and use the quantified measurements to optimize the design of the controller.To demonstrate the effectiveness of our method,we deploy the trained policy on hardware and,with a single control policy,the quadruped robot can perform various learned skills,including automatic gait transitions,high kick,and forward jump.

Inferring object properties from human interaction and transferring them to new motions

Humans regularly interact with their surrounding objects.Such interactions often result in strongly correlated motions between humans and the interacting objects.We thus ask:"Is it possible to infer object properties from skeletal motion alone,even without seeing the interacting object itself?"In this paper,we present a fine-grained action recognition method that learns to infer such latent object properties from human interaction motion alone.This inference allows us to disentangle the motion from the object property and transfer object properties to a given motion.We collected a large number of videos and 3 D skeletal motions of performing actors using an inertial motion capture device.We analyzed similar actions and learned subtle differences between them to reveal latent properties of the interacting objects.In particular,we learned to identify the interacting object,by estimating its weight,or its spillability.Our results clearly demonstrate that motions and interacting objects are highly correlated and that related object latent properties can be inferred from 3 D skeleton sequences alone,leading to new synthesis possibilities for motions involving human interaction.Our dataset is available at http://vcc.szu.edu.cn/research/2020/IT.html.

A Survey on Human Performance Capture and Animation

With the rapid development of computing technology, three-dimensional （3D） human body models and their dynamic motions are widely used in the digital entertainment industry. Human performance mainly involves human body shapes and motions. Key research problems in human performance animation include how to capture and analyze static geometric appearance and dynamic movement of human bodies, and how to simulate human body motions with physical effects. In this survey, according to the main research directions of human body performance capture and animation, we summarize recent advances in key research topics, namely human body surface reconstruction, motion capture and synthesis, as well as physics-based motion simulation, and further discuss future research problems and directions. We hope this will be helpful for readers to have a comprehensive understanding of human performance capture and animation.