基于图文注意力融合的主题标签推荐

Multimodal Hashtag Recommendation Based on Image and Text Attention Fusion

为了解决社交媒体平台上的信息超载问题,帮助用户快速捕捉所需信息,对基于多模态内容的标签推荐问题进行研究。针对不同模态间的异质性差异,采用共注意力机制进行跨模态内容的特征建模与融合;针对多标签分类方法只能推荐出数据集标签空间中标签的不足,采用Seq2Seq框架生成新的标签序列,并通过一种聚合策略将分类方法的推荐结果聚合到生成的标签序列中,得到2种方法的统一推荐模型。在大规模数据集上的实验结果表明:多模态方法比单模态方法更具优势,所提出的统一推荐模型的F1值比仅使用单模态的对比模型高9.44百分点;生成新标签序列的方法也优于传统的分类方法,所提出的标签序列生成模型的F1值比对比模型COA高3.41百分点;所提出的统一推荐模型UNIFIED-CO-ATT的F1值比GEN-CO-ATT模型高1.25百分点,其效果优于其他对比模型。所提出的模型综合了分类方法和生成方法的特点,可以使推荐的标签同时具有准确性和新颖性。

In order to solve the information overload problem on social media platforms and help users quickly capture the required information,in this study the problem of hashtag recommendation based on multimodal content was investigated.To address the heterogeneous differences between different modalities,a co-attention mechanism was used to model and fuse features of cross-modal content,and use Seq2Seq framework was used to generate new hashtag sequences to address the deficiency that multi-label classification methods could only recommend hashtags in the hashtag space of the dataset.An aggregation strategy was used to aggregate the recommendation results of classification methods into the generated hashtag sequences to obtain a unified recommendation model for both methods.The experimental results on a large-scale dataset showed that,firstly,the multimodal approach was more advantageous than the unimodal approach,and the unified recommendation model proposed in this paper had 9.44 percentage points improvement in F1 value over the comparison model using unimodal approach,and 3.41 percentage points improvement over the comparison model using the classification method.Finally,the unified recommendation model UNIFIED-CO-ATT is 1.25 percentage points higher than GEN-CO-ATT in F1 values.The model proposed in this study could combine the advantages of classification and generation methods and could make the recommended hashtags have the advantages of accuracy and novelty at the same time.