In this work,we present a reconfigurable data glove design to capture different modes of human hand-object interactions,which are critical in training embodied artificial intelligence(AI)agents for fine manipulation t...In this work,we present a reconfigurable data glove design to capture different modes of human hand-object interactions,which are critical in training embodied artificial intelligence(AI)agents for fine manipulation tasks.To achieve various downstream tasks with distinct features,our reconfigurable data glove operates in three modes sharing a unified backbone design that reconstructs hand gestures in real time.In the tactile-sensing mode,the glove system aggregates manipulation force via customized force sensors made from a soft and thin piezoresistive material;this design minimizes interference during complex hand movements.The virtual reality(VR)mode enables real-time interaction in a physically plausible fashion:A caging-based approach is devised to determine stable grasps by detecting collision events.Leveraging a state-of-the-art finite element method,the simulation mode collects data on fine-grained four-dimensionalmanipulation events comprising hand and object motions in three-dimensional space and how the object's physical properties(e.g.,stress and energy)change in accordance with manipulation over time.Notably,the glove system presented here is the first to use high-fidelity simulation to investigate the unobservable physical and causal factors behind manipulation actions.In a series of experiments,we characterize our data glove in terms of individual sensors and the overall sys-tem.More specifically,we evaluate the system's three modes by①recording hand gestures and associ-ated forces,②improving manipulation fluency in VR,and③producing realistic simulation effects of various tool uses,respectively.Based on these three modes,our reconfigurable data glove collects and reconstructs fine-grained human grasp data in both physical and virtual environments,thereby opening up new avenues for the learning of manipulation skills for embodied Al agents.展开更多
基金the National Key Research and Development Program of China(2021ZD0150200)the Beijing Nova Program.
文摘In this work,we present a reconfigurable data glove design to capture different modes of human hand-object interactions,which are critical in training embodied artificial intelligence(AI)agents for fine manipulation tasks.To achieve various downstream tasks with distinct features,our reconfigurable data glove operates in three modes sharing a unified backbone design that reconstructs hand gestures in real time.In the tactile-sensing mode,the glove system aggregates manipulation force via customized force sensors made from a soft and thin piezoresistive material;this design minimizes interference during complex hand movements.The virtual reality(VR)mode enables real-time interaction in a physically plausible fashion:A caging-based approach is devised to determine stable grasps by detecting collision events.Leveraging a state-of-the-art finite element method,the simulation mode collects data on fine-grained four-dimensionalmanipulation events comprising hand and object motions in three-dimensional space and how the object's physical properties(e.g.,stress and energy)change in accordance with manipulation over time.Notably,the glove system presented here is the first to use high-fidelity simulation to investigate the unobservable physical and causal factors behind manipulation actions.In a series of experiments,we characterize our data glove in terms of individual sensors and the overall sys-tem.More specifically,we evaluate the system's three modes by①recording hand gestures and associ-ated forces,②improving manipulation fluency in VR,and③producing realistic simulation effects of various tool uses,respectively.Based on these three modes,our reconfigurable data glove collects and reconstructs fine-grained human grasp data in both physical and virtual environments,thereby opening up new avenues for the learning of manipulation skills for embodied Al agents.
