多模态数据处理是一个重要的研究领域,它可以通过结合文本、图像等多种信息来提高模型性能.然而,由于不同模态之间的异构性以及信息融合的挑战,设计有效的多模态分类模型仍然是一个具有挑战性的问题.本文提出了一种新的多模态分类模型—...多模态数据处理是一个重要的研究领域,它可以通过结合文本、图像等多种信息来提高模型性能.然而,由于不同模态之间的异构性以及信息融合的挑战,设计有效的多模态分类模型仍然是一个具有挑战性的问题.本文提出了一种新的多模态分类模型——MCM-ICE,它通过联合独立编码和协同编码策略来解决特征表示和特征融合的挑战.MCM-ICE在Fashion-Gen和Hateful Memes Challenge两个数据集上进行了实验,结果表明该模型在这两项任务中均优于现有的最先进方法.本文还探究了协同编码模块Transformer输出层的不同向量选取对结果的影响,结果表明选取[CLS]向量和去除[CLS]的向量的平均池化向量可以获得最佳结果.消融研究和探索性分析支持了MCM-ICE模型在处理多模态分类任务方面的有效性.展开更多
Mutual information is widely used in medical image registration, because it does not require preprocessing the image. However, the local maximum problem in the registration is insurmountable. We combine mutual informa...Mutual information is widely used in medical image registration, because it does not require preprocessing the image. However, the local maximum problem in the registration is insurmountable. We combine mutual information and gradient information to solve this problem and apply it to the non-rigid deformation image registration. To improve the accuracy, we provide some implemental issues, for example, the Powell searching algorithm, gray interpolation and consideration of outlier points. The experimental results show the accuracy of the method and the feasibility in non-rigid medical image registration.展开更多
Laser cleaning is a highly nonlinear physical process for solving poor single-modal(e.g., acoustic or vision)detection performance and low inter-information utilization. In this study, a multi-modal feature fusion net...Laser cleaning is a highly nonlinear physical process for solving poor single-modal(e.g., acoustic or vision)detection performance and low inter-information utilization. In this study, a multi-modal feature fusion network model was constructed based on a laser paint removal experiment. The alignment of heterogeneous data under different modals was solved by combining the piecewise aggregate approximation and gramian angular field. Moreover, the attention mechanism was introduced to optimize the dual-path network and dense connection network, enabling the sampling characteristics to be extracted and integrated. Consequently, the multi-modal discriminant detection of laser paint removal was realized. According to the experimental results, the verification accuracy of the constructed model on the experimental dataset was 99.17%, which is 5.77% higher than the optimal single-modal detection results of the laser paint removal. The feature extraction network was optimized by the attention mechanism, and the model accuracy was increased by 3.3%. Results verify the improved classification performance of the constructed multi-modal feature fusion model in detecting laser paint removal, the effective integration of acoustic data and visual image data, and the accurate detection of laser paint removal.展开更多
文摘多模态数据处理是一个重要的研究领域,它可以通过结合文本、图像等多种信息来提高模型性能.然而,由于不同模态之间的异构性以及信息融合的挑战,设计有效的多模态分类模型仍然是一个具有挑战性的问题.本文提出了一种新的多模态分类模型——MCM-ICE,它通过联合独立编码和协同编码策略来解决特征表示和特征融合的挑战.MCM-ICE在Fashion-Gen和Hateful Memes Challenge两个数据集上进行了实验,结果表明该模型在这两项任务中均优于现有的最先进方法.本文还探究了协同编码模块Transformer输出层的不同向量选取对结果的影响,结果表明选取[CLS]向量和去除[CLS]的向量的平均池化向量可以获得最佳结果.消融研究和探索性分析支持了MCM-ICE模型在处理多模态分类任务方面的有效性.
文摘Mutual information is widely used in medical image registration, because it does not require preprocessing the image. However, the local maximum problem in the registration is insurmountable. We combine mutual information and gradient information to solve this problem and apply it to the non-rigid deformation image registration. To improve the accuracy, we provide some implemental issues, for example, the Powell searching algorithm, gray interpolation and consideration of outlier points. The experimental results show the accuracy of the method and the feasibility in non-rigid medical image registration.
基金Project(51875491) supported by the National Natural Science Foundation of ChinaProject(2021T3069) supported by the Fujian Science and Technology Plan STS Project,China。
文摘Laser cleaning is a highly nonlinear physical process for solving poor single-modal(e.g., acoustic or vision)detection performance and low inter-information utilization. In this study, a multi-modal feature fusion network model was constructed based on a laser paint removal experiment. The alignment of heterogeneous data under different modals was solved by combining the piecewise aggregate approximation and gramian angular field. Moreover, the attention mechanism was introduced to optimize the dual-path network and dense connection network, enabling the sampling characteristics to be extracted and integrated. Consequently, the multi-modal discriminant detection of laser paint removal was realized. According to the experimental results, the verification accuracy of the constructed model on the experimental dataset was 99.17%, which is 5.77% higher than the optimal single-modal detection results of the laser paint removal. The feature extraction network was optimized by the attention mechanism, and the model accuracy was increased by 3.3%. Results verify the improved classification performance of the constructed multi-modal feature fusion model in detecting laser paint removal, the effective integration of acoustic data and visual image data, and the accurate detection of laser paint removal.
