In the immersive flow visualization based on virtual reality,how to meet the needs of complex professional flow visualization analysis by natural human–computer interaction is a pressing problem.In order to achieve t...In the immersive flow visualization based on virtual reality,how to meet the needs of complex professional flow visualization analysis by natural human–computer interaction is a pressing problem.In order to achieve the natural and efficient human–computer interaction,we analyze the interaction requirements of flow visualization and study the characteristics of four human–computer interaction channels:hand,head,eye and voice.We give out some multimodal interaction design suggestions and then propose three multimodal interaction methods:head&hand,head&hand&eye and head&hand&eye&voice.The freedom of gestures,the stability of the head,the convenience of eyes and the rapid retrieval of voices are used to improve the accuracy and efficiency of interaction.The interaction load is balanced by multimodal interaction to reduce fatigue.The evaluation shows that our multimodal interaction has higher accuracy,faster time efficiency and much lower fatigue than the traditional joystick interaction.展开更多
Immersive environments have become increasingly popular for visualizing and exploring large-scale,complex scientific data because of their key features:immersion,engagement,and awareness.Virtual reality offers numerou...Immersive environments have become increasingly popular for visualizing and exploring large-scale,complex scientific data because of their key features:immersion,engagement,and awareness.Virtual reality offers numerous new interaction possibilities,including tactile and tangible interactions,gestures,and voice commands.However,it is crucial to determine the most effective combination of these techniques for a more natural interaction experience.In this paper,we present MEinVR,a novel multimodal interaction technique for exploring 3D molecular data in virtual reality.MEinVR combines VR controller and voice input to provide a more intuitive way for users to manipulate data in immersive environments.By using the VR controller to select locations and regions of interest and voice commands to perform tasks,users can efficiently perform complex data exploration tasks.Our findings provide suggestions for the design of multimodal interaction techniques in 3D data exploration in virtual reality.展开更多
Biography videos based on life performances of prominent figures in history aim to describe great mens' life.In this paper,a novel interactive video summarization for biography video based on multimodal fusion is ...Biography videos based on life performances of prominent figures in history aim to describe great mens' life.In this paper,a novel interactive video summarization for biography video based on multimodal fusion is proposed,which is a novel approach of visualizing the specific features for biography video and interacting with video content by taking advantage of the ability of multimodality.In general,a story of movie progresses by dialogues of characters and the subtitles are produced with the basis on the dialogues which contains all the information related to the movie.In this paper,JGibbsLDA is applied to extract key words from subtitles because the biography video consists of different aspects to depict the characters' whole life.In terms of fusing keywords and key-frames,affinity propagation is adopted to calculate the similarity between each key-frame cluster and keywords.Through the method mentioned above,a video summarization is presented based on multimodal fusion which describes video content more completely.In order to reduce the time spent on searching the interest video content and get the relationship between main characters,a kind of map is adopted to visualize video content and interact with video summarization.An experiment is conducted to evaluate video summarization and the results demonstrate that this system can formally facilitate the exploration of video content while improving interaction and finding events of interest efficiently.展开更多
Teleoperation is of great importance in the area of robotics,especially when people are unavailable in the robot workshop.It provides a way for people to control robots remotely using human intelligence.In this paper,...Teleoperation is of great importance in the area of robotics,especially when people are unavailable in the robot workshop.It provides a way for people to control robots remotely using human intelligence.In this paper,a robotic teleoperation system for precise robotic manipulation is established.The data glove and the 7-degrees of freedom(DOFs)force feedback controller are used for the remote control interaction.The control system and the monitor system are designed for the remote precise manipulation.The monitor system contains an image acquisition system and a human-machine interaction module,and aims to simulate and detect the robot running state.Besides,a visual object tracking algorithm is developed to estimate the states of the dynamic system from noisy observations.The established robotic teleoperation systemis applied to a series of experiments,and high-precision results are obtained,showing the effectiveness of the physical system.展开更多
In the majority of the interaction process, the operator often focuses on the tracked 3D hand gesture model at the ＂interaction points＂ in the collision detectionscene, such as ＂grasp＂ and ＂release＂ and objects ...In the majority of the interaction process, the operator often focuses on the tracked 3D hand gesture model at the ＂interaction points＂ in the collision detectionscene, such as ＂grasp＂ and ＂release＂ and objects in the scene, without paying attention to the tracked 3D hand gesture model in the total procedure. Thus in this paper, a visual attention distribution model of operator in the ＂grasp＂, ＂translation＂, ＂release＂ and other basic operation procedures is first studied and a 3D hand gesture tracking algorithm based on this distribution model is proposed. Utilizing the algorithm, in the period with a low degree of visual attention, a pre-stored 3D hand gesture animation can be used to directly visualise a 3D hand gesture model in the interactive scene; in the time period with a high degree of visual attention, an existing ＂frame-by-frame tracking＂ approach can be adopted to obtain a 3D gesture model. The results demonstrate that the proposed method can achieve real-time tracking of 3D hand gestures with an effective improvement on the efficiency, fluency, and availability of 3D hand gesture interaction.展开更多
基金supported in part by the National Natural Science Foundation of China(No.61872304,No.61802320)the State Key Laboratory of Aerodynamics(SKLA20200203)the National Numerical Windtunnel Project(NNW2019ZT6-A17).
文摘In the immersive flow visualization based on virtual reality,how to meet the needs of complex professional flow visualization analysis by natural human–computer interaction is a pressing problem.In order to achieve the natural and efficient human–computer interaction,we analyze the interaction requirements of flow visualization and study the characteristics of four human–computer interaction channels:hand,head,eye and voice.We give out some multimodal interaction design suggestions and then propose three multimodal interaction methods:head&hand,head&hand&eye and head&hand&eye&voice.The freedom of gestures,the stability of the head,the convenience of eyes and the rapid retrieval of voices are used to improve the accuracy and efficiency of interaction.The interaction load is balanced by multimodal interaction to reduce fatigue.The evaluation shows that our multimodal interaction has higher accuracy,faster time efficiency and much lower fatigue than the traditional joystick interaction.
基金NSFC,PR China(62272396)XJTLU Research Development Funding,PR China RDF-19-02-11.
文摘Immersive environments have become increasingly popular for visualizing and exploring large-scale,complex scientific data because of their key features:immersion,engagement,and awareness.Virtual reality offers numerous new interaction possibilities,including tactile and tangible interactions,gestures,and voice commands.However,it is crucial to determine the most effective combination of these techniques for a more natural interaction experience.In this paper,we present MEinVR,a novel multimodal interaction technique for exploring 3D molecular data in virtual reality.MEinVR combines VR controller and voice input to provide a more intuitive way for users to manipulate data in immersive environments.By using the VR controller to select locations and regions of interest and voice commands to perform tasks,users can efficiently perform complex data exploration tasks.Our findings provide suggestions for the design of multimodal interaction techniques in 3D data exploration in virtual reality.
基金Supported by the National Key Research and Development Plan(2016YFB1001200)the Natural Science Foundation of China(U1435220,61232013)Natural Science Research Projects of Universities in Jiangsu Province(16KJA520003)
文摘Biography videos based on life performances of prominent figures in history aim to describe great mens' life.In this paper,a novel interactive video summarization for biography video based on multimodal fusion is proposed,which is a novel approach of visualizing the specific features for biography video and interacting with video content by taking advantage of the ability of multimodality.In general,a story of movie progresses by dialogues of characters and the subtitles are produced with the basis on the dialogues which contains all the information related to the movie.In this paper,JGibbsLDA is applied to extract key words from subtitles because the biography video consists of different aspects to depict the characters' whole life.In terms of fusing keywords and key-frames,affinity propagation is adopted to calculate the similarity between each key-frame cluster and keywords.Through the method mentioned above,a video summarization is presented based on multimodal fusion which describes video content more completely.In order to reduce the time spent on searching the interest video content and get the relationship between main characters,a kind of map is adopted to visualize video content and interact with video summarization.An experiment is conducted to evaluate video summarization and the results demonstrate that this system can formally facilitate the exploration of video content while improving interaction and finding events of interest efficiently.
基金NSFC-Shenzhen Robotics Research Center Project(No.U2013207)the Beijing Science and Technology Plan Project(No.Z191100008019008)。
文摘Teleoperation is of great importance in the area of robotics,especially when people are unavailable in the robot workshop.It provides a way for people to control robots remotely using human intelligence.In this paper,a robotic teleoperation system for precise robotic manipulation is established.The data glove and the 7-degrees of freedom(DOFs)force feedback controller are used for the remote control interaction.The control system and the monitor system are designed for the remote precise manipulation.The monitor system contains an image acquisition system and a human-machine interaction module,and aims to simulate and detect the robot running state.Besides,a visual object tracking algorithm is developed to estimate the states of the dynamic system from noisy observations.The established robotic teleoperation systemis applied to a series of experiments,and high-precision results are obtained,showing the effectiveness of the physical system.
基金Supported by the National Natural Science Foundation of China(61472163)the National Key Research&Development Plan of China(2016YFB1001403)the Science and Technology Project of Shandong Province(2015GGX101025)
文摘In the majority of the interaction process, the operator often focuses on the tracked 3D hand gesture model at the ＂interaction points＂ in the collision detectionscene, such as ＂grasp＂ and ＂release＂ and objects in the scene, without paying attention to the tracked 3D hand gesture model in the total procedure. Thus in this paper, a visual attention distribution model of operator in the ＂grasp＂, ＂translation＂, ＂release＂ and other basic operation procedures is first studied and a 3D hand gesture tracking algorithm based on this distribution model is proposed. Utilizing the algorithm, in the period with a low degree of visual attention, a pre-stored 3D hand gesture animation can be used to directly visualise a 3D hand gesture model in the interactive scene; in the time period with a high degree of visual attention, an existing ＂frame-by-frame tracking＂ approach can be adopted to obtain a 3D gesture model. The results demonstrate that the proposed method can achieve real-time tracking of 3D hand gestures with an effective improvement on the efficiency, fluency, and availability of 3D hand gesture interaction.
