A new dynamic terminal sliding mode control (DTSMC) technique is proposed for a class of single-input and single-output (SISO) uncertain nonlinear systems. The dynamic terminal sliding mode controller is formulate...A new dynamic terminal sliding mode control (DTSMC) technique is proposed for a class of single-input and single-output (SISO) uncertain nonlinear systems. The dynamic terminal sliding mode controller is formulated based on Lyapunov theory such that the existence of the sliding phase of the closed-loop control system can be guaranteed, chattering phenomenon caused by the switching control action can be eliminated, and high precision performance is realized. Moreover, by designing terminal equation, the output tracking error converges to zero in finite time, the reaching phase of DSMC is eliminated and global robustness is obtained. The simulation results for an inverted pendulum are given to demonstrate the properties of the proposed method.展开更多
With the advance of smart material science,robotics is evolving from rigid robots to soft robots.Compared to rigid robots,soft robots can safely interact with the environment,easily navigate in unstructured fields,and...With the advance of smart material science,robotics is evolving from rigid robots to soft robots.Compared to rigid robots,soft robots can safely interact with the environment,easily navigate in unstructured fields,and be minimized to operate in narrow spaces,owning to the new actuation and sensing technologies developed by the smart materials.In the review,different actuation and sensing technologies based on different smart materials are analyzed and summarized.According to the driving or feedback signals,actuators are categorized into electrically responsive actuators,thermally responsive actuators,magnetically responsive actuators,and photoresponsive actuators;sensors are categorized into resistive sensors,capacitive sensors,magnetic sensors,and optical waveguide sensors.After introducing the principle and several robotic prototypes of some typical materials in each category of the actuators and sensors.The advantages and disadvantages of the actuators and sensors are compared based on the categories,and their potential applications in robotics are also presented.展开更多
A neuro-fuzzy system model based on automatic fuzzy dustering is proposed. A hybrid model identification algorithm is also developed to decide the model structure and model parameters. The algorithm mainly includes th...A neuro-fuzzy system model based on automatic fuzzy dustering is proposed. A hybrid model identification algorithm is also developed to decide the model structure and model parameters. The algorithm mainly includes three parts:1) Automatic fuzzy C-means (AFCM), which is applied to generate fuzzy rttles automatically, and then fix on the size of the neuro-fuzzy network, by which the complexity of system design is reducesd greatly at the price of the fitting capability; 2) R.ecursive least square estimation (RLSE). It is used to update the parameters of Takagi-Sugeno model, which is employed to describe the behavior of the system;3) Gradient descent algorithm is also proposed for the fuzzy values according to the back propagation algorithm of neural network. Finally,modeling the dynamical equation of the two-link manipulator with the proposed approach is illustrated to validate the feasibility of the method.展开更多
A novel fuzzy terminal sliding mode control (FTSMC) scheme is proposed for position tracking of a class of second-order nonlinear uncertain system. In the proposed scheme, we integrate input-output linearization tec...A novel fuzzy terminal sliding mode control (FTSMC) scheme is proposed for position tracking of a class of second-order nonlinear uncertain system. In the proposed scheme, we integrate input-output linearization technique to cancel the nonlinearities. By using a function-augmented sliding hyperplane, it is guaranteed that the output tracking error converges to zero in finite time which can be set arbitrarily. The proposed scheme eliminates reaching phase problem, so that the closed-loop system always shows invariance property to parameter uncertainties. Fuzzy logic systems are used to approximate the unknown system functions and switch item. Robust adaptive law is proposed to reduce approximation errors between true nonlinear functions and fuzzy systems, thus chattering phenomenon can be eliminated. Stability of the proposed control scheme is proved and the scheme is applied to an inverted pendulum system. Simulation studies are provided to confirm performance and effectiveness of the proposed control approach.展开更多
The control of a high Degree of Freedom(DoF) robot to grasp a target in three-dimensional space using Brain-Computer Interface(BCI) remains a very difficult problem to solve. Design of synchronous BCI requires the use...The control of a high Degree of Freedom(DoF) robot to grasp a target in three-dimensional space using Brain-Computer Interface(BCI) remains a very difficult problem to solve. Design of synchronous BCI requires the user perform the brain activity task all the time according to the predefined paradigm; such a process is boring and fatiguing. Furthermore, the strategy of switching between robotic auto-control and BCI control is not very reliable because the accuracy of Motor Imagery(MI) pattern recognition rarely reaches 100%. In this paper, an asynchronous BCI shared control method is proposed for the high DoF robotic grasping task. The proposed method combines BCI control and automatic robotic control to simultaneously consider the robotic vision feedback and revise the unreasonable control commands. The user can easily mentally control the system and is only required to intervene and send brain commands to the automatic control system at the appropriate time according to the experience of the user. Two experiments are designed to validate our method: one aims to illustrate the accuracy of MI pattern recognition of our asynchronous BCI system; the other is the online practical experiment that controls the robot to grasp a target while avoiding an obstacle using the asynchronous BCI shared control method that can improve the safety and robustness of our system.展开更多
This paper focuses on multi-modal Information Perception(IP)for Soft Robotic Hands(SRHs)using Machine Learning(ML)algorithms.A flexible Optical Fiber-based Curvature Sensor(OFCS)is fabricated,consisting of a Light-Emi...This paper focuses on multi-modal Information Perception(IP)for Soft Robotic Hands(SRHs)using Machine Learning(ML)algorithms.A flexible Optical Fiber-based Curvature Sensor(OFCS)is fabricated,consisting of a Light-Emitting Diode(LED),photosensitive detector,and optical fiber.Bending the roughened optical fiber generates lower light intensity,which reflecting the curvature of the soft finger.Together with the curvature and pressure information,multi-modal IP is performed to improve the recognition accuracy.Recognitions of gesture,object shape,size,and weight are implemented with multiple ML approaches,including the Supervised Learning Algorithms(SLAs)of K-Nearest Neighbor(KNN),Support Vector Machine(SVM),Logistic Regression(LR),and the unSupervised Learning Algorithm(un-SLA)of K-Means Clustering(KMC).Moreover,Optical Sensor Information(OSI),Pressure Sensor Information(PSI),and Double-Sensor Information(DSI)are adopted to compare the recognition accuracies.The experiment results demonstrate that the proposed sensors and recognition approaches are feasible and effective.The recognition accuracies obtained using the above ML algorithms and three modes of sensor information are higer than 85 percent for almost all combinations.Moreover,DSI is more accurate when compared to single modal sensor information and the KNN algorithm with a DSI outperforms the other combinations in recognition accuracy.展开更多
With the accelerated aging of the global population and escalating labor costs, more service robots are needed to help people perform complex tasks. As such, human-robot interaction is a particularly important researc...With the accelerated aging of the global population and escalating labor costs, more service robots are needed to help people perform complex tasks. As such, human-robot interaction is a particularly important research topic. To effectively transfer human behavior skills to a robot, in this study, we conveyed skill-learning functions via our proposed wearable device. The robotic teleoperation system utilizes interactive demonstration via the wearable device by directly controlling the speed of the motors. We present a rotation-invariant dynamicalmovement-primitive method for learning interaction skills. We also conducted robotic teleoperation demonstrations and designed imitation learning experiments. The experimental human-robot interaction results confirm the effectiveness of the proposed method.展开更多
This study investigates the consensus problem of second-order multi-agent systems subject to time- varying interval-like delays. The notion of consensus is extended to networks containing antagonistic interactions mod...This study investigates the consensus problem of second-order multi-agent systems subject to time- varying interval-like delays. The notion of consensus is extended to networks containing antagonistic interactions modeled by negative weights on the communication graph. A unified framework is established to address both the stationary and dynamic consensus issues in sampled-data settings. Using the reciprocally convex approach, a sufficient condition for consensus is derived in terms of matrix inequalities. Numerical examples are provided to illustrate the effectiveness of the proposed result.展开更多
In this paper, the attitude control problem of rigid body is addressed with considering inertia uncertainty,bounded time-varying disturbances, angular velocity-free measurement, and unknown non-symmetric saturation in...In this paper, the attitude control problem of rigid body is addressed with considering inertia uncertainty,bounded time-varying disturbances, angular velocity-free measurement, and unknown non-symmetric saturation input. Using a mathematical transformation, the effects of bounded time-varying disturbances, uncertain inertia,and saturation input are combined as total disturbances. A novel finite-time observer is designed to estimate the unknown angular velocity and the total disturbances. For attitude control, an observer-based sliding-mode control protocol is proposed to force the system state convergence to the desired sliding-mode surface; the finite-time stability is guaranteed via Lyapunov theory analysis. Finally, a numerical simulation is presented to illustrate the effective performance of the proposed sliding-mode control protocol.展开更多
Robot recognition tasks usually require multiple homogeneous or heterogeneous sensors which intrinsically generate sequential, redundant, and storage demanding data with various noise pollution. Thus, online machine l...Robot recognition tasks usually require multiple homogeneous or heterogeneous sensors which intrinsically generate sequential, redundant, and storage demanding data with various noise pollution. Thus, online machine learning algorithms performing efficient sensory feature fusion have become a hot topic in robot recognition domain. This paper proposes an online multi-kernel extreme learning machine (OM-ELM) which assembles multiple ELM classifiers and optimizes the kernel weights with a p-norm formulation of multi-kernel learning (MKL) problem. It can be applied in feature fusion applications that require incremental learning over multiple sequential sensory readings. The performance of OM-ELM is tested towards four different robot recognition tasks. By comparing to several state-of-the-art online models for multi-kernel learning, we claim that our method achieves a superior or equivalent training accuracy and generalization ability with less training time. Practical suggestions are also given to aid effective online fusion of robot sensory features.展开更多
In this work, we use a deep learning method to tackle the Zero-Shot Learning(ZSL) problem in tactile material recognition by incorporating the advanced semantic information into a training model. Our main technical co...In this work, we use a deep learning method to tackle the Zero-Shot Learning(ZSL) problem in tactile material recognition by incorporating the advanced semantic information into a training model. Our main technical contribution is our proposal of an end-to-end deep learning framework for solving the tactile ZSL problem. In this framework, we use a Convolutional Neural Network(CNN) to extract the spatial features and Long Short-Term Memory(LSTM) to extract the temporal features in dynamic tactile sequences, and develop a loss function suitable for the ZSL setting. We present the results of experimental evaluations on publicly available datasets, which show the effectiveness of the proposed method.展开更多
A brain-computer interface(BCI) system can recognize the mental activities pattern by computer algorithms to control the external devices. Electroencephalogram(EEG) is one of the most common used approach for BCI due ...A brain-computer interface(BCI) system can recognize the mental activities pattern by computer algorithms to control the external devices. Electroencephalogram(EEG) is one of the most common used approach for BCI due to the convenience and non-invasive implement. Therefore, more and more BCIs have been designed for the disabled people that suffer from stroke or spinal cord injury to help them for rehabilitation and life. We introduce the common BCI paradigms, the signal processing, and feature extraction methods. Then, we survey the different combined modes of hybrids BCIs and review the design of the synchronous/asynchronous BCIs.Finally, the shared control methods are discussed.展开更多
Various living creatures exhibit embodiment intelligence,which is reflected by a collaborative interaction of the brain,body,and environment.The actual behavior of embodiment intelligence is generated by a continuous ...Various living creatures exhibit embodiment intelligence,which is reflected by a collaborative interaction of the brain,body,and environment.The actual behavior of embodiment intelligence is generated by a continuous and dynamic interaction between a subject and the environment through information perception and physical manipulation.The physical interaction between a robot and the environment is the basis for realizing embodied perception and learning.Tactile information plays a critical role in this physical interaction process.It can be used to ensure safety,stability,and compliance,and can provide unique information that is difficult to capture using other perception modalities.However,due to the limitations of existing sensors and perception and learning methods,the development of robotic tactile research lags significantly behind other sensing modalities,such as vision and hearing,thereby seriously restricting the development of robotic embodiment intelligence.This paper presents the current challenges related to robotic tactile embodiment intelligence and reviews the theory and methods of robotic embodied tactile intelligence.Tactile perception and learning methods for embodiment intelligence can be designed based on the development of new large-scale tactile array sensing devices,with the aim to make breakthroughs in the neuromorphic computing technology of tactile intelligence.展开更多
Modern computational models have leveraged biological advances in human brain research. This study addresses the problem of multimodal learning with the help of brain-inspired models. Specifically, a unified multimoda...Modern computational models have leveraged biological advances in human brain research. This study addresses the problem of multimodal learning with the help of brain-inspired models. Specifically, a unified multimodal learning architecture is proposed based on deep neural networks, which are inspired by the biology of the visual cortex of the human brain. This unified framework is validated by two practical multimodal learning tasks: image captioning, involving visual and natural language signals, and visual-haptic fusion, involving haptic and visual signals. Extensive experiments are conducted under the framework, and competitive results are achieved.展开更多
基金the National Natural Science Foundation of China (No.60474025, 90405017).
文摘A new dynamic terminal sliding mode control (DTSMC) technique is proposed for a class of single-input and single-output (SISO) uncertain nonlinear systems. The dynamic terminal sliding mode controller is formulated based on Lyapunov theory such that the existence of the sliding phase of the closed-loop control system can be guaranteed, chattering phenomenon caused by the switching control action can be eliminated, and high precision performance is realized. Moreover, by designing terminal equation, the output tracking error converges to zero in finite time, the reaching phase of DSMC is eliminated and global robustness is obtained. The simulation results for an inverted pendulum are given to demonstrate the properties of the proposed method.
基金Supported by National Key Research and Development Program of China(Grant No.2019YFB 1309800)National Natural Science Foundation of China(Grant Nos.62173197,91848206)Beijing Science&Technology Project(Grant No.Z191100008019008).
文摘With the advance of smart material science,robotics is evolving from rigid robots to soft robots.Compared to rigid robots,soft robots can safely interact with the environment,easily navigate in unstructured fields,and be minimized to operate in narrow spaces,owning to the new actuation and sensing technologies developed by the smart materials.In the review,different actuation and sensing technologies based on different smart materials are analyzed and summarized.According to the driving or feedback signals,actuators are categorized into electrically responsive actuators,thermally responsive actuators,magnetically responsive actuators,and photoresponsive actuators;sensors are categorized into resistive sensors,capacitive sensors,magnetic sensors,and optical waveguide sensors.After introducing the principle and several robotic prototypes of some typical materials in each category of the actuators and sensors.The advantages and disadvantages of the actuators and sensors are compared based on the categories,and their potential applications in robotics are also presented.
文摘A neuro-fuzzy system model based on automatic fuzzy dustering is proposed. A hybrid model identification algorithm is also developed to decide the model structure and model parameters. The algorithm mainly includes three parts:1) Automatic fuzzy C-means (AFCM), which is applied to generate fuzzy rttles automatically, and then fix on the size of the neuro-fuzzy network, by which the complexity of system design is reducesd greatly at the price of the fitting capability; 2) R.ecursive least square estimation (RLSE). It is used to update the parameters of Takagi-Sugeno model, which is employed to describe the behavior of the system;3) Gradient descent algorithm is also proposed for the fuzzy values according to the back propagation algorithm of neural network. Finally,modeling the dynamical equation of the two-link manipulator with the proposed approach is illustrated to validate the feasibility of the method.
基金This work was supported by the National Natural Science Foundation of China (No. 60474025, 90405017).
文摘A novel fuzzy terminal sliding mode control (FTSMC) scheme is proposed for position tracking of a class of second-order nonlinear uncertain system. In the proposed scheme, we integrate input-output linearization technique to cancel the nonlinearities. By using a function-augmented sliding hyperplane, it is guaranteed that the output tracking error converges to zero in finite time which can be set arbitrarily. The proposed scheme eliminates reaching phase problem, so that the closed-loop system always shows invariance property to parameter uncertainties. Fuzzy logic systems are used to approximate the unknown system functions and switch item. Robust adaptive law is proposed to reduce approximation errors between true nonlinear functions and fuzzy systems, thus chattering phenomenon can be eliminated. Stability of the proposed control scheme is proved and the scheme is applied to an inverted pendulum system. Simulation studies are provided to confirm performance and effectiveness of the proposed control approach.
基金supported by the National Natural Science Foundation of China (Nos. 91420302 and 91520201)Innovation Cultivating Fund Project 17 163 12 ZT 001 019 01
文摘The control of a high Degree of Freedom(DoF) robot to grasp a target in three-dimensional space using Brain-Computer Interface(BCI) remains a very difficult problem to solve. Design of synchronous BCI requires the user perform the brain activity task all the time according to the predefined paradigm; such a process is boring and fatiguing. Furthermore, the strategy of switching between robotic auto-control and BCI control is not very reliable because the accuracy of Motor Imagery(MI) pattern recognition rarely reaches 100%. In this paper, an asynchronous BCI shared control method is proposed for the high DoF robotic grasping task. The proposed method combines BCI control and automatic robotic control to simultaneously consider the robotic vision feedback and revise the unreasonable control commands. The user can easily mentally control the system and is only required to intervene and send brain commands to the automatic control system at the appropriate time according to the experience of the user. Two experiments are designed to validate our method: one aims to illustrate the accuracy of MI pattern recognition of our asynchronous BCI system; the other is the online practical experiment that controls the robot to grasp a target while avoiding an obstacle using the asynchronous BCI shared control method that can improve the safety and robustness of our system.
基金support provided by the National Natural Science Foundation of China (Nos. 61803267 and 61572328)the China Postdoctoral Science Foundation (No.2017M622757)+1 种基金the Beijing Science and Technology program (No.Z171100000817007)the National Science Foundation of China (NSFC) and the German Re-search Foundation (DFG) in the project Cross Modal Learning,NSFC 61621136008/DFG TRR-169
文摘This paper focuses on multi-modal Information Perception(IP)for Soft Robotic Hands(SRHs)using Machine Learning(ML)algorithms.A flexible Optical Fiber-based Curvature Sensor(OFCS)is fabricated,consisting of a Light-Emitting Diode(LED),photosensitive detector,and optical fiber.Bending the roughened optical fiber generates lower light intensity,which reflecting the curvature of the soft finger.Together with the curvature and pressure information,multi-modal IP is performed to improve the recognition accuracy.Recognitions of gesture,object shape,size,and weight are implemented with multiple ML approaches,including the Supervised Learning Algorithms(SLAs)of K-Nearest Neighbor(KNN),Support Vector Machine(SVM),Logistic Regression(LR),and the unSupervised Learning Algorithm(un-SLA)of K-Means Clustering(KMC).Moreover,Optical Sensor Information(OSI),Pressure Sensor Information(PSI),and Double-Sensor Information(DSI)are adopted to compare the recognition accuracies.The experiment results demonstrate that the proposed sensors and recognition approaches are feasible and effective.The recognition accuracies obtained using the above ML algorithms and three modes of sensor information are higer than 85 percent for almost all combinations.Moreover,DSI is more accurate when compared to single modal sensor information and the KNN algorithm with a DSI outperforms the other combinations in recognition accuracy.
基金supported by the National Natural Science Foundation of China (Nos. 61503212, 61473089, U1613212, and 61327809)the Beijing Science and Technology Program (No. Z171100000817007)+1 种基金the German Research Foundation (DFG) in project Cross Modal Learning (No. NSFC 61621136008/DFG TRR169)the Suzhou Special Program (No. 2016SZ0219)
文摘With the accelerated aging of the global population and escalating labor costs, more service robots are needed to help people perform complex tasks. As such, human-robot interaction is a particularly important research topic. To effectively transfer human behavior skills to a robot, in this study, we conveyed skill-learning functions via our proposed wearable device. The robotic teleoperation system utilizes interactive demonstration via the wearable device by directly controlling the speed of the motors. We present a rotation-invariant dynamicalmovement-primitive method for learning interaction skills. We also conducted robotic teleoperation demonstrations and designed imitation learning experiments. The experimental human-robot interaction results confirm the effectiveness of the proposed method.
基金jointly supported by the Specialized Research Fund for the Doctoral Program of Higher Education (No. 20110002110015)the National Key Basic Research and Development (973) Program of China (No. 2012CB821206)+1 种基金the National National Science Foundation of China (Nos. 61473161, 61174069, and 51374082)Tsinghua University Initiative Scientific Research Program
文摘This study investigates the consensus problem of second-order multi-agent systems subject to time- varying interval-like delays. The notion of consensus is extended to networks containing antagonistic interactions modeled by negative weights on the communication graph. A unified framework is established to address both the stationary and dynamic consensus issues in sampled-data settings. Using the reciprocally convex approach, a sufficient condition for consensus is derived in terms of matrix inequalities. Numerical examples are provided to illustrate the effectiveness of the proposed result.
基金supported by the National Natural Science Foundation of China (No. 61403399)
文摘In this paper, the attitude control problem of rigid body is addressed with considering inertia uncertainty,bounded time-varying disturbances, angular velocity-free measurement, and unknown non-symmetric saturation input. Using a mathematical transformation, the effects of bounded time-varying disturbances, uncertain inertia,and saturation input are combined as total disturbances. A novel finite-time observer is designed to estimate the unknown angular velocity and the total disturbances. For attitude control, an observer-based sliding-mode control protocol is proposed to force the system state convergence to the desired sliding-mode surface; the finite-time stability is guaranteed via Lyapunov theory analysis. Finally, a numerical simulation is presented to illustrate the effective performance of the proposed sliding-mode control protocol.
文摘Robot recognition tasks usually require multiple homogeneous or heterogeneous sensors which intrinsically generate sequential, redundant, and storage demanding data with various noise pollution. Thus, online machine learning algorithms performing efficient sensory feature fusion have become a hot topic in robot recognition domain. This paper proposes an online multi-kernel extreme learning machine (OM-ELM) which assembles multiple ELM classifiers and optimizes the kernel weights with a p-norm formulation of multi-kernel learning (MKL) problem. It can be applied in feature fusion applications that require incremental learning over multiple sequential sensory readings. The performance of OM-ELM is tested towards four different robot recognition tasks. By comparing to several state-of-the-art online models for multi-kernel learning, we claim that our method achieves a superior or equivalent training accuracy and generalization ability with less training time. Practical suggestions are also given to aid effective online fusion of robot sensory features.
基金supported in part by the National Natural Science Foundation of China (Nos. 61673238, 61703284, and 61327809)the Beijing Municipal Science and Technology Commission (No. D171100005017002)
文摘In this work, we use a deep learning method to tackle the Zero-Shot Learning(ZSL) problem in tactile material recognition by incorporating the advanced semantic information into a training model. Our main technical contribution is our proposal of an end-to-end deep learning framework for solving the tactile ZSL problem. In this framework, we use a Convolutional Neural Network(CNN) to extract the spatial features and Long Short-Term Memory(LSTM) to extract the temporal features in dynamic tactile sequences, and develop a loss function suitable for the ZSL setting. We present the results of experimental evaluations on publicly available datasets, which show the effectiveness of the proposed method.
基金supported by the Innovation Cultivating Fund Project(No.17-163-12-ZT-001-019-01)
文摘A brain-computer interface(BCI) system can recognize the mental activities pattern by computer algorithms to control the external devices. Electroencephalogram(EEG) is one of the most common used approach for BCI due to the convenience and non-invasive implement. Therefore, more and more BCIs have been designed for the disabled people that suffer from stroke or spinal cord injury to help them for rehabilitation and life. We introduce the common BCI paradigms, the signal processing, and feature extraction methods. Then, we survey the different combined modes of hybrids BCIs and review the design of the synchronous/asynchronous BCIs.Finally, the shared control methods are discussed.
基金supported by the National Natural Science Foundation of China under Grant No.61703284 and Grant No.61673238
文摘Various living creatures exhibit embodiment intelligence,which is reflected by a collaborative interaction of the brain,body,and environment.The actual behavior of embodiment intelligence is generated by a continuous and dynamic interaction between a subject and the environment through information perception and physical manipulation.The physical interaction between a robot and the environment is the basis for realizing embodied perception and learning.Tactile information plays a critical role in this physical interaction process.It can be used to ensure safety,stability,and compliance,and can provide unique information that is difficult to capture using other perception modalities.However,due to the limitations of existing sensors and perception and learning methods,the development of robotic tactile research lags significantly behind other sensing modalities,such as vision and hearing,thereby seriously restricting the development of robotic embodiment intelligence.This paper presents the current challenges related to robotic tactile embodiment intelligence and reviews the theory and methods of robotic embodied tactile intelligence.Tactile perception and learning methods for embodiment intelligence can be designed based on the development of new large-scale tactile array sensing devices,with the aim to make breakthroughs in the neuromorphic computing technology of tactile intelligence.
基金supported by National Natural Science Foundation of China(Grant Nos.61621136008,61327809,61210013,91420302,and 91520201)
文摘Modern computational models have leveraged biological advances in human brain research. This study addresses the problem of multimodal learning with the help of brain-inspired models. Specifically, a unified multimodal learning architecture is proposed based on deep neural networks, which are inspired by the biology of the visual cortex of the human brain. This unified framework is validated by two practical multimodal learning tasks: image captioning, involving visual and natural language signals, and visual-haptic fusion, involving haptic and visual signals. Extensive experiments are conducted under the framework, and competitive results are achieved.