TECMH:Transformer-Based Cross-Modal Hashing For Fine-Grained Image-Text Retrieval

In recent years,cross-modal hash retrieval has become a popular research field because of its advantages of high efficiency and low storage.Cross-modal retrieval technology can be applied to search engines,crossmodalmedical processing,etc.The existing main method is to use amulti-label matching paradigm to finish the retrieval tasks.However,such methods do not use fine-grained information in the multi-modal data,which may lead to suboptimal results.To avoid cross-modal matching turning into label matching,this paper proposes an end-to-end fine-grained cross-modal hash retrieval method,which can focus more on the fine-grained semantic information of multi-modal data.First,the method refines the image features and no longer uses multiple labels to represent text features but uses BERT for processing.Second,this method uses the inference capabilities of the transformer encoder to generate global fine-grained features.Finally,in order to better judge the effect of the fine-grained model,this paper uses the datasets in the image text matching field instead of the traditional label-matching datasets.This article experiment on Microsoft COCO(MS-COCO)and Flickr30K datasets and compare it with the previous classicalmethods.The experimental results show that this method can obtain more advanced results in the cross-modal hash retrieval field.

ViT2CMH:Vision Transformer Cross-Modal Hashing for Fine-Grained Vision-Text Retrieval

In recent years,the development of deep learning has further improved hash retrieval technology.Most of the existing hashing methods currently use Convolutional Neural Networks(CNNs)and Recurrent Neural Networks(RNNs)to process image and text information,respectively.This makes images or texts subject to local constraints,and inherent label matching cannot capture finegrained information,often leading to suboptimal results.Driven by the development of the transformer model,we propose a framework called ViT2CMH mainly based on the Vision Transformer to handle deep Cross-modal Hashing tasks rather than CNNs or RNNs.Specifically,we use a BERT network to extract text features and use the vision transformer as the image network of the model.Finally,the features are transformed into hash codes for efficient and fast retrieval.We conduct extensive experiments on Microsoft COCO(MS-COCO)and Flickr30K,comparing with baselines of some hashing methods and image-text matching methods,showing that our method has better performance.

Adequate alignment and interaction for cross-modal retrieval

Background Cross-modal retrieval has attracted widespread attention in many cross-media similarity search applications,particularly image-text retrieval in the fields of computer vision and natural language processing.Recently,visual and semantic embedding(VSE)learning has shown promising improvements in image text retrieval tasks.Most existing VSE models employ two unrelated encoders to extract features and then use complex methods to contextualize and aggregate these features into holistic embeddings.Despite recent advances,existing approaches still suffer from two limitations:(1)without considering intermediate interactions and adequate alignment between different modalities,these models cannot guarantee the discriminative ability of representations;and(2)existing feature aggregators are susceptible to certain noisy regions,which may lead to unreasonable pooling coefficients and affect the quality of the final aggregated features.Methods To address these challenges,we propose a novel cross-modal retrieval model containing a well-designed alignment module and a novel multimodal fusion encoder that aims to learn the adequate alignment and interaction of aggregated features to effectively bridge the modality gap.Results Experiments on the Microsoft COCO and Flickr30k datasets demonstrated the superiority of our model over state-of-the-art methods.

Cross-Modal Hashing Retrieval Based on Deep Residual Network

In the era of big data rich inWe Media,the single mode retrieval system has been unable to meet people’s demand for information retrieval.This paper proposes a new solution to the problem of feature extraction and unified mapping of different modes:A Cross-Modal Hashing retrieval algorithm based on Deep Residual Network(CMHR-DRN).The model construction is divided into two stages:The first stage is the feature extraction of different modal data,including the use of Deep Residual Network(DRN)to extract the image features,using the method of combining TF-IDF with the full connection network to extract the text features,and the obtained image and text features used as the input of the second stage.In the second stage,the image and text features are mapped into Hash functions by supervised learning,and the image and text features are mapped to the common binary Hamming space.In the process of mapping,the distance measurement of the original distance measurement and the common feature space are kept unchanged as far as possible to improve the accuracy of Cross-Modal Retrieval.In training the model,adaptive moment estimation(Adam)is used to calculate the adaptive learning rate of each parameter,and the stochastic gradient descent(SGD)is calculated to obtain the minimum loss function.The whole training process is completed on Caffe deep learning framework.Experiments show that the proposed algorithm CMHR-DRN based on Deep Residual Network has better retrieval performance and stronger advantages than other Cross-Modal algorithms CMFH,CMDN and CMSSH.

An Efficient Encrypted Speech Retrieval Based on Unsupervised Hashing and B+ Tree Dynamic Index

Existing speech retrieval systems are frequently confronted with expanding volumes of speech data.The dynamic updating strategy applied to construct the index can timely process to add or remove unnecessary speech data to meet users’real-time retrieval requirements.This study proposes an efficient method for retrieving encryption speech,using unsupervised deep hashing and B+ tree dynamic index,which avoid privacy leak-age of speech data and enhance the accuracy and efficiency of retrieval.The cloud’s encryption speech library is constructed by using the multi-threaded Dijk-Gentry-Halevi-Vaikuntanathan(DGHV)Fully Homomorphic Encryption(FHE)technique,which encrypts the original speech.In addition,this research employs Residual Neural Network18-Gated Recurrent Unit(ResNet18-GRU),which is used to learn the compact binary hash codes,store binary hash codes in the designed B+tree index table,and create a mapping relation of one to one between the binary hash codes and the corresponding encrypted speech.External B+tree index technology is applied to achieve dynamic index updating of the B+tree index table,thereby satisfying users’needs for real-time retrieval.The experimental results on THCHS-30 and TIMIT showed that the retrieval accuracy of the proposed method is more than 95.84%compared to the existing unsupervised hashing methods.The retrieval efficiency is greatly improved.Compared to the method of using hash index tables,and the speech data’s security is effectively guaranteed.

Fusion of Hash-Based Hard and Soft Biometrics for Enhancing Face Image Database Search and Retrieval

The utilization of digital picture search and retrieval has grown substantially in numerous fields for different purposes during the last decade,owing to the continuing advances in image processing and computer vision approaches.In multiple real-life applications,for example,social media,content-based face picture retrieval is a well-invested technique for large-scale databases,where there is a significant necessity for reliable retrieval capabilities enabling quick search in a vast number of pictures.Humans widely employ faces for recognizing and identifying people.Thus,face recognition through formal or personal pictures is increasingly used in various real-life applications,such as helping crime investigators retrieve matching images from face image databases to identify victims and criminals.However,such face image retrieval becomes more challenging in large-scale databases,where traditional vision-based face analysis requires ample additional storage space than the raw face images already occupied to store extracted lengthy feature vectors and takes much longer to process and match thousands of face images.This work mainly contributes to enhancing face image retrieval performance in large-scale databases using hash codes inferred by locality-sensitive hashing(LSH)for facial hard and soft biometrics as(Hard BioHash)and(Soft BioHash),respectively,to be used as a search input for retrieving the top-k matching faces.Moreover,we propose the multi-biometric score-level fusion of both face hard and soft BioHashes(Hard-Soft BioHash Fusion)for further augmented face image retrieval.The experimental outcomes applied on the Labeled Faces in the Wild(LFW)dataset and the related attributes dataset(LFW-attributes),demonstrate that the retrieval performance of the suggested fusion approach(Hard-Soft BioHash Fusion)significantly improved the retrieval performance compared to solely using Hard BioHash or Soft BioHash in isolation,where the suggested method provides an augmented accuracy of 87%when executed on 1000 specimens and 77%on 5743 samples.These results remarkably outperform the results of the Hard BioHash method by(50%on the 1000 samples and 30%on the 5743 samples),and the Soft BioHash method by(78%on the 1000 samples and 63%on the 5743 samples).

Secure Content Based Image Retrieval Scheme Based on Deep Hashing and Searchable Encryption

To solve the problem that the existing ciphertext domain image retrieval system is challenging to balance security,retrieval efficiency,and retrieval accuracy.This research suggests a searchable encryption and deep hashing-based secure image retrieval technique that extracts more expressive image features and constructs a secure,searchable encryption scheme.First,a deep learning framework based on residual network and transfer learn-ing model is designed to extract more representative image deep features.Secondly,the central similarity is used to quantify and construct the deep hash sequence of features.The Paillier homomorphic encryption encrypts the deep hash sequence to build a high-security and low-complexity searchable index.Finally,according to the additive homomorphic property of Paillier homomorphic encryption,a similarity measurement method suitable for com-puting in the retrieval system’s security is ensured by the encrypted domain.The experimental results,which were obtained on Web Image Database from the National University of Singapore(NUS-WIDE),Microsoft Common Objects in Context(MS COCO),and ImageNet data sets,demonstrate the system’s robust security and precise retrieval,the proposed scheme can achieve efficient image retrieval without revealing user privacy.The retrieval accuracy is improved by at least 37%compared to traditional hashing schemes.At the same time,the retrieval time is saved by at least 9.7%compared to the latest deep hashing schemes.

Lung Nodule Image Retrieval Based on Convolutional Neural Networks and Hashing

Lung medical image retrieval based on content similarity plays an important role in computer-aided diagnosis of lung cancer.In recent years,binary hashing has become a hot topic in this field due to its compressed storage and fast query speed.Traditional hashing methods often rely on highdimensional features based hand-crafted methods,which might not be optimally compatible with lung nodule images.Also,different hashing bits contribute to the image retrieval differently,and therefore treating the hashing bits equally affects the retrieval accuracy.Hence,an image retrieval method of lung nodule images is proposed with the basis on convolutional neural networks and hashing.First,apre-trained and fine-tuned convolutional neural network is employed to learn multilevel semantic features of the lung nodules.Principal components analysis is utilized to remove redundant information and preserve informative semantic features of the lung nodules.Second,the proposed method relies on nine sign labels of lung nodules for the training set,and the semantic feature is combined to construct hashing functions.Finally,returned lung nodule images can be easily ranked with the query-adaptive search method based on weighted Hamming distance.Extensive experiments and evaluations on the dataset demonstrate that the proposed method can significantly improve the expression ability of lung nodule images,which further validates the effectiveness of the proposed method.

IDSH: An Improved Deep Supervised Hashing Method for Image Retrieval

Image retrieval has become more and more important because of the explosive growth of images on the Internet.Traditional image retrieval methods have limited image retrieval performance due to the poor image expression abhility of visual feature and high dimension of feature.Hashing is a widely-used method for Approximate Nearest Neighbor(ANN)search due to its rapidity and timeliness.Meanwhile,Convolutional Neural Networks(CNNs)have strong discriminative characteristics which are used for image classification.In this paper,we propose a CNN architecture based on improved deep supervised hashing(IDSH)method,by which the binary compact codes can be generated directly.The main contributions of this paper are as follows:first,we add a Batch Normalization(BN)layer before each activation layer to prevent the gradient from vanishing and improve the training speed;secondly,we use Divide-and-Encode Module to map image features to approximate hash codes;finally,we adopt center loss to optimize training.Extensive experimental results on four large-scale datasets:MNIST,CIFAR-10,NUS-WIDE and SVHN demonstrate the effectiveness of the proposed method compared with other state-of-the-art hashing methods.

Multi-Index Image Retrieval Hash Algorithm Based on Multi-View Feature Coding

In recent years,with the massive growth of image data,how to match the image required by users quickly and efficiently becomes a challenge.Compared with single-view feature,multi-view feature is more accurate to describe image information.The advantages of hash method in reducing data storage and improving efficiency also make us study how to effectively apply to large-scale image retrieval.In this paper,a hash algorithm of multi-index image retrieval based on multi-view feature coding is proposed.By learning the data correlation between different views,this algorithm uses multi-view data with deeper level image semantics to achieve better retrieval results.This algorithm uses a quantitative hash method to generate binary sequences,and uses the hash code generated by the association features to construct database inverted index files,so as to reduce the memory burden and promote the efficient matching.In order to reduce the matching error of hash code and ensure the retrieval accuracy,this algorithm uses inverted multi-index structure instead of single-index structure.Compared with other advanced image retrieval method,this method has better retrieval performance.

Image Retrieval Using Deep Convolutional Neural Networks and Regularized Locality Preserving Indexing Strategy

Convolutional Neural Networks (CNN) has been a very popular area in large scale data processing and many works have demonstrate that CNN is a very promising tool in many field, e.g., image classification and image retrieval. Theoretically, CNN features can become better and better with the increase of CNN layers. But on the other side more layers can dramatically increase the computational cost on the same condition of other devices. In addition to CNN features, how to dig out the potential information contained in the features is also an important aspect. In this paper, we propose a novel approach utilize deep CNN to extract image features and then introduce a Regularized Locality Preserving Indexing (RLPI) method which can make features more differentiated through learning a new space of the data space. First, we apply deep networks (VGG-net) to extract image features and then introduce Regularized Locality Preserving Indexing (RLPI) method to train a model. Finally, the new feature space can be generated through this model and then can be used to image retrieval.

基于Vision Transformer Hashing的民族布艺图案哈希检索算法研究

针对民族布艺图案复杂多样、语义提取、图像识别与检索困难等问题,以蜡染图案和织锦图案为例,提出一种图像检索算法,以提高匹配和检索民族布艺图案的准确度。结合民族布艺图案领域知识将民族布艺图案图像进行预处理,使用VIT为主干网络在哈希检索算法框架下进行图像检索。该方法优化了深度哈希检索算法,通过自身的自注意力机制提升了提取图案深层语义特征的能力,提高了深度哈希算法检索民族布艺图案的速度和精度。实验结果表明:提出的方法最佳检索精度可以达到95.32%。

Robust cross-modal retrieval with alignment refurbishment

Cross-modal retrieval tries to achieve mutual retrieval between modalities by establishing consistent alignment for different modal data.Currently,many cross-modal retrieval methods have been proposed and have achieved excellent results;however,these are trained with clean cross-modal pairs,which are semantically matched but costly,compared with easily available data with noise alignment(i.e.,paired but mismatched in semantics).When training these methods with noise-aligned data,the performance degrades dramatically.Therefore,we propose a robust cross-modal retrieval with alignment refurbishment(RCAR),which significantly reduces the impact of noise on the model.Specifically,RCAR first conducts multi-task learning to slow down the overfitting to the noise to make data separable.Then,RCAR uses a two-component beta-mixture model to divide them into clean and noise alignments and refurbishes the label according to the posterior probability of the noise-alignment component.In addition,we define partial and complete noises in the noise-alignment paradigm.Experimental results show that,compared with the popular cross-modal retrieval methods,RCAR achieves more robust performance with both types of noise.

文本语义哈希技术研究进展

文本语义哈希是在满足语义相似性约束下将文本转化为低维二值数据的神经编码技术,支持基于汉明距离的高效检索,以解决有限计算资源约束下海量文本的相似性计算问题。文本语义哈希技术存在诸多挑战,包括如何在低维二值编码中融入类别信息、如何丰富编码的语义信息以提升模型鲁棒性、如何解决离散输出的模型梯度估计等关键问题。文章首先综述文本语义哈希任务的重要研究发展,详细讨论了无监督文本语义哈希模型和融合类别信息的有监督文本语义哈希模型的技术细节,分析基于近邻文本、隐式主题等信息的语义增强技术以及模型优化等关键技术;然后,综述文本语义哈希任务相关数据集和评估指标,对比了各类文本语义哈希技术的特点和性能;最后,讨论了文本语义哈希技术的未来发展方向。

基于Transformer生成对抗网络的跨模态哈希检索算法

考虑生成对抗网络在保持跨模态数据之间的流形结构的优势,并结合Transformer利用自注意力和无须使用卷积的优点,提出一种基于Transformer生成对抗网络的跨模态哈希检索算法。首先在ImageNet数据集上预训练Vision Transformer框架,并将其作为图像特征提取的主干网络,然后将不同模态的数据分割为共享特征和私有特征。接着,构建对抗学习模块减少不同模态的共享特征的分布距离与保持语义一致性,同时增大不同模态的私有特征分布距离与保持语义非一致性。最后将通用的特征表示映射为紧凑的哈希码,实现跨模态哈希检索。实验结果表明,在公共数据集上,所提算法优于对比算法。

基于图像检索技术的受载煤样裂隙演化规律

在煤岩试样破坏过程中进行计算机断层(computed tomography,CT)扫描测试,可实现煤岩试样内部裂隙结构可视化表征,但是在加载过程中,因试样位移变形会引起CT扫描层定位误差,进而影响试验结果准确性。为了解决这一问题,利用图像相似性检索技术,检索出受载试样在不同应力水平下的最相似的CT图像,从而更加准确地描述受载煤样裂隙演化规律。研究结果表明:在三轴压缩破坏过程中,试样应力应变曲线分为5个明显的阶段,即初始压密阶段、线弹性变形阶段、塑性屈服阶段、峰值破坏阶段以及残余变形阶段。基于感知哈希算法的图像检索技术能够准确识别并定位不同应力阶段相似度最高的CT图像,且CT图像相似度随着轴向荷载不断增大而逐渐降低。在整个三轴加载过程中,CT图像的相似度绝对差均值表现出两个明显的变化阶段:缓慢增加阶段和快速增加阶段。分形维数可以定量描述裂隙演化,受载样品破坏过程中分形维数主要经历了缓慢减小、缓慢增大和快速增大3个阶段。

基于图卷积的无监督跨模态哈希检索算法

为解决当前无监督跨模态哈希检索在全局相似性矩阵构建和异构数据语义信息融合中存在的困难,提出一种基于图卷积的无监督跨模态哈希检索算法(GCUH)。采用分层次聚合的方式,将各个模态的相似性结构编码到全局相似性矩阵中,获得跨模态的成对相似性信息来指导学习。使用图卷积模块融合跨模态信息,消除邻居结构中的噪声干扰,形成完备的跨模态表征,提出两种相似性保持的损失函数约束哈希码的一致性。与基线模型相比,GCUH在NUS-WIDE数据集上使用64位哈希码执行文本检索图片任务的检索精度提升了6.3%。

基于注意力机制和软匹配的多标签遥感图像检索方法

针对卷积神经网络对于多标签遥感图像特征提取能力弱、不能准确反映遥感图像多标签复杂性的问题,提出基于注意力机制和软匹配的多标签遥感图像检索方法。在特征提取阶段,以密集卷积神经网络模型为基础,在每个密集块(Dense Block)后添加CBAM(Convolutional Block Attention Module)层,实现对多标签图像区域特征提取。在模型训练时,利用区分硬匹配与软匹配的联合损失函数,学习图像的哈希编码表示。通过评估遥感图像哈希编码间的汉明距离,实现相似图像的检索。实验结果表明,所提方法在数据集NUS-WIDE和多标签遥感图像数据集DLRSD上与其他基于全局特征的深度哈希方法相比,明显提升了检索准确率。

基于哈希学习算法的专业课程资源库安全检索方法

建筑管理专业具有课程资源多样、专业性强以及不断更新变化的特性,导致在资源库进行检索时的mAP值较低。为此,提出了一种基于哈希学习算法的专业课程资源库安全检索方法。该方法系统地整理、分类并描述课程资源,形成建筑管理专业课程资源本体,建立检索索引,并编码存储资源信息、属性及关联。通过哈希学习算法提取特征,生成哈希函数,将高维数据映射为低维哈希码,从而建立检索模型。当用户查询时,系统能够快速定位相关资源,并高效、安全地返回结果,实现了专业课程库的安全检索。实验结果表明,该方法能够适应课程资源的特性,使在检索时的mAP值较高,提升了用户体验,同时确保了课程资源库的安全性和稳定性。