Toward Fine-grained Image Retrieval with Adaptive Deep Learning for Cultural Heritage Image

Fine-grained image classification is a challenging research topic because of the high degree of similarity among categories and the high degree of dissimilarity for a specific category caused by different poses and scales.A cul-tural heritage image is one of thefine-grained images because each image has the same similarity in most cases.Using the classification technique,distinguishing cultural heritage architecture may be difficult.This study proposes a cultural heri-tage content retrieval method using adaptive deep learning forfine-grained image retrieval.The key contribution of this research was the creation of a retrieval mod-el that could handle incremental streams of new categories while maintaining its past performance in old categories and not losing the old categorization of a cul-tural heritage image.The goal of the proposed method is to perform a retrieval task for classes.Incremental learning for new classes was conducted to reduce the re-training process.In this step,the original class is not necessary for re-train-ing which we call an adaptive deep learning technique.Cultural heritage in the case of Thai archaeological site architecture was retrieved through machine learn-ing and image processing.We analyze the experimental results of incremental learning forfine-grained images with images of Thai archaeological site architec-ture from world heritage provinces in Thailand,which have a similar architecture.Using afine-grained image retrieval technique for this group of cultural heritage images in a database can solve the problem of a high degree of similarity among categories and a high degree of dissimilarity for a specific category.The proposed method for retrieving the correct image from a database can deliver an average accuracy of 85 percent.Adaptive deep learning forfine-grained image retrieval was used to retrieve cultural heritage content,and it outperformed state-of-the-art methods infine-grained image retrieval.

Fine-Grained Classification of Remote Sensing Ship Images Based on Improved VAN

The remote sensing ships’fine-grained classification technology makes it possible to identify certain ship types in remote sensing images,and it has broad application prospects in civil and military fields.However,the current model does not examine the properties of ship targets in remote sensing images with mixed multi-granularity features and a complicated backdrop.There is still an opportunity for future enhancement of the classification impact.To solve the challenges brought by the above characteristics,this paper proposes a Metaformer and Residual fusion network based on Visual Attention Network(VAN-MR)for fine-grained classification tasks.For the complex background of remote sensing images,the VAN-MR model adopts the parallel structure of large kernel attention and spatial attention to enhance the model’s feature extraction ability of interest targets and improve the classification performance of remote sensing ship targets.For the problem of multi-grained feature mixing in remote sensing images,the VAN-MR model uses a Metaformer structure and a parallel network of residual modules to extract ship features.The parallel network has different depths,considering both high-level and lowlevel semantic information.The model achieves better classification performance in remote sensing ship images with multi-granularity mixing.Finally,the model achieves 88.73%and 94.56%accuracy on the public fine-grained ship collection-23(FGSC-23)and FGSCR-42 datasets,respectively,while the parameter size is only 53.47 M,the floating point operations is 9.9 G.The experimental results show that the classification effect of VAN-MR is superior to that of traditional CNNs model and visual model with Transformer structure under the same parameter quantity.

Fine-Grained Features for Image Captioning

Image captioning involves two different major modalities(image and sentence)that convert a given image into a language that adheres to visual semantics.Almost all methods first extract image features to reduce the difficulty of visual semantic embedding and then use the caption model to generate fluent sentences.The Convolutional Neural Network(CNN)is often used to extract image features in image captioning,and the use of object detection networks to extract region features has achieved great success.However,the region features retrieved by this method are object-level and do not pay attention to fine-grained details because of the detection model’s limitation.We offer an approach to address this issue that more properly generates captions by fusing fine-grained features and region features.First,we extract fine-grained features using a panoramic segmentation algorithm.Second,we suggest two fusion methods and contrast their fusion outcomes.An X-linear Attention Network(X-LAN)serves as the foundation for both fusion methods.According to experimental findings on the COCO dataset,the two-branch fusion approach is superior.It is important to note that on the COCO Karpathy test split,CIDEr is increased up to 134.3%in comparison to the baseline,highlighting the potency and viability of our method.

Learning Noise-Assisted Robust Image Features for Fine-Grained Image Retrieval

Fine-grained image search is one of the most challenging tasks in computer vision that aims to retrieve similar images at the fine-grained level for a given query image.The key objective is to learn discriminative fine-grained features by training deep models such that similar images are clustered,and dissimilar images are separated in the low embedding space.Previous works primarily focused on defining local structure loss functions like triplet loss,pairwise loss,etc.However,training via these approaches takes a long training time,and they have poor accuracy.Additionally,representations learned through it tend to tighten up in the embedded space and lose generalizability to unseen classes.This paper proposes a noise-assisted representation learning method for fine-grained image retrieval to mitigate these issues.In the proposed work,class manifold learning is performed in which positive pairs are created with noise insertion operation instead of tightening class clusters.And other instances are treated as negatives within the same cluster.Then a loss function is defined to penalize when the distance between instances of the same class becomes too small relative to the noise pair in that class in embedded space.The proposed approach is validated on CARS-196 and CUB-200 datasets and achieved better retrieval results(85.38%recall@1 for CARS-196%and 70.13%recall@1 for CUB-200)compared to other existing methods.

Fine-grained Ship Image Recognition Based on BCNN with Inception and AM-Softmax

The fine-grained ship image recognition task aims to identify various classes of ships.However,small inter-class,large intra-class differences between ships,and lacking of training samples are the reasons that make the task difficult.Therefore,to enhance the accuracy of the fine-grained ship image recognition,we design a fine-grained ship image recognition network based on bilinear convolutional neural network(BCNN)with Inception and additive margin Softmax(AM-Softmax).This network improves the BCNN in two aspects.Firstly,by introducing Inception branches to the BCNN network,it is helpful to enhance the ability of extracting comprehensive features from ships.Secondly,by adding margin values to the decision boundary,the AM-Softmax function can better extend the inter-class differences and reduce the intra-class differences.In addition,as there are few publicly available datasets for fine-grained ship image recognition,we construct a Ship-43 dataset containing 47,300 ship images belonging to 43 categories.Experimental results on the constructed Ship-43 dataset demonstrate that our method can effectively improve the accuracy of ship image recognition,which is 4.08%higher than the BCNN model.Moreover,comparison results on the other three public fine-grained datasets(Cub,Cars,and Aircraft)further validate the effectiveness of the proposed method.

Fine-Grained Emotion Prediction for Movie and Television scene images

For the task of content retrieval,analysis and generation of film and television scene images in the field of intelligent editing,fine-grained emotion recognition and prediction of images is of great significance.In this paper,the fusion of traditional perceptual features,art features and multi-channel deep learning features are used to reflect the emotion expression of different levels of the image.In addition,the integrated learning model with stacking architecture based on linear regression coefficient and sentiment correlations,which is called the LS-stacking model,is proposed according to the factor association between multi-dimensional emotions.The experimental results prove that the mixed feature and LS-stacking model can predict well on the 16 emotion categories of the self-built image dataset.This study improves the fine-grained recognition ability of image emotion by computers,which helps to increase the intelligence and automation degree of visual retrieval and post-production system.

Bedding parallel fractures in fine-grained sedimentary rocks:Recognition,formation mechanisms,and prediction using well log

Core,thin section,conventional and image logs are used to provide insights into distribution of fractures in fine grained sedimentary rocks of Permian Lucaogou Formation in Jimusar Sag.Bedding parallel fractures are common in fine grained sedimentary rocks which are characterized by layered structures.Core and thin section analysis reveal that fractures in Lucaogou Formation include tectonic inclined fracture,bedding parallel fracture,and abnormal high pressure fracture.Bedding parallel fractures are abundant,but only minor amounts of them remain open,and most of them are partly to fully sealed by carbonate minerals(calcite)and bitumen.Bedding parallel fractures result in a rapid decrease in resistivity,and they are recognized on image logs to extend along bedding planes and have discontinuous surfaces due to partly-fully filled resistive carbonate minerals as well as late stage dissolution.A comprehensive interpretation of distribution of bedding parallel fractures is performed with green line,red line,yellow line and blue line representing bedding planes,induced fractures,resistive fractures,and open(bedding and inclined)fractures,respectively.The strike of bedding parallel fractures is coinciding with bedding planes.Bedding parallel fractures are closely associated with the amounts of bedding planes,and high density of bedding planes favor the formation of bedding parallel fractures.Alternating dark and bright layers have the most abundant bedding parallel fractures on the image logs,and the bedding parallel fractures are always associated with low resistivity zones.The results above may help optimize sweet spots in fine grained sedimentary rocks,and improve future fracturing design and optimize well spacing.

Prediction of multiscale laminae structure and reservoir quality in fine-grained sedimentary rocks:The Permian Lucaogou Formation in Jimusar Sag,Junggar Basin

Fine-grained sedimentary rocks have become a research focus as important reservoirs and source rocks for tight and shale oil and gas.Laminae development determines the accumulation and production of tight and shale oil and gas in fine-grained rocks.However,due to the resolution limit of conventional logs,it is challenging to recognize the features of centimeter-scale laminae.To close this gap,complementary studies,including core observation,thin section,X-ray diffraction(XRD),conventional log analysis,and slabs of image logs,were conducted to unravel the centimeter-scale laminae.The laminae recognition models were built using well logs.The fine-grained rocks can be divided into laminated rocks(lamina thickness of<0.01 m),layered rocks(0.01-0.1 m),and massive rocks(no layer or layer spacing of>0.1 m)according to the laminae scale from core observations.According to the mineral superposition assemblages from thin-section observations,the laminated rocks can be further divided into binary,ternary,and multiple structures.The typical mineral components,slabs,and T2spectrum distributions of various lamina types are unraveled.The core can identify the centimeter-millimeter-scale laminae,and the thin section can identify the millimeter-micrometer-scale laminae.Furthermore,they can detect mineral types and their superposition sequence.Conventional logs can identify the meter-scale layers,whereas image logs and related slabs can identify the laminae variations at millimeter-centimeter scales.Therefore,the slab of image logs combined with thin sections can identify laminae assemblage characteristics,including the thickness and vertical assemblage.The identification and classification of lamina structure of various scales on a single well can be predicted using conventional logs,image logs,and slabs combined with thin sections.The layered rocks have better reservoir quality and oil-bearing potential than the massive and laminated rocks.The laminated rocks’binary lamina is better than the ternary and multiple layers due to the high content of felsic minerals.The abovementioned results build the prediction model for multiscale laminae structure using well logs,helping sweet spots prediction in the Permian Lucaogou Formation in the Jimusar Sag and fine-grained sedimentary rocks worldwide.

Light-Weight Selective Image Encryption for Privacy Preservation

To protect personal privacy and confidential preservation,access control is used to authorize legal users for safe browsing the authorized contents on photos.The access control generates an authorization rule according to each permission assignment.However,the general access control is inappropriate to apply in some social services(e.g.,photos posted on Flickr and Instagram,personal image management in mobile phone) because of the increasing popularity of digital images being stored and managed.With low maintenance loads,this paper integrates the data hiding technique to propose an access control mechanism for privacy preservation.The proposed scheme changes the partial regions of a given image as random pads (called selective image encryption) and only allows the authorized people to remedy the random pads back to meaningful ones which are with similar visual qualities of original ones.

Attention Guided Food Recognition via Multi-Stage Local Feature Fusion

The task of food image recognition,a nuanced subset of fine-grained image recognition,grapples with substantial intra-class variation and minimal inter-class differences.These challenges are compounded by the irregular and multi-scale nature of food images.Addressing these complexities,our study introduces an advanced model that leverages multiple attention mechanisms and multi-stage local fusion,grounded in the ConvNeXt architecture.Our model employs hybrid attention(HA)mechanisms to pinpoint critical discriminative regions within images,substantially mitigating the influence of background noise.Furthermore,it introduces a multi-stage local fusion(MSLF)module,fostering long-distance dependencies between feature maps at varying stages.This approach facilitates the assimilation of complementary features across scales,significantly bolstering the model’s capacity for feature extraction.Furthermore,we constructed a dataset named Roushi60,which consists of 60 different categories of common meat dishes.Empirical evaluation of the ETH Food-101,ChineseFoodNet,and Roushi60 datasets reveals that our model achieves recognition accuracies of 91.12%,82.86%,and 92.50%,respectively.These figures not only mark an improvement of 1.04%,3.42%,and 1.36%over the foundational ConvNeXt network but also surpass the performance of most contemporary food image recognition methods.Such advancements underscore the efficacy of our proposed model in navigating the intricate landscape of food image recognition,setting a new benchmark for the field.

Multi-granularity sequence generation for hierarchical image classification

Hierarchical multi-granularity image classification is a challenging task that aims to tag each given image with multiple granularity labels simultaneously.Existing methods tend to overlook that different image regions contribute differently to label prediction at different granularities,and also insufficiently consider relationships between the hierarchical multi-granularity labels.We introduce a sequence-to-sequence mechanism to overcome these two problems and propose a multi-granularity sequence generation(MGSG)approach for the hierarchical multi-granularity image classification task.Specifically,we introduce a transformer architecture to encode the image into visual representation sequences.Next,we traverse the taxonomic tree and organize the multi-granularity labels into sequences,and vectorize them and add positional information.The proposed multi-granularity sequence generation method builds a decoder that takes visual representation sequences and semantic label embedding as inputs,and outputs the predicted multi-granularity label sequence.The decoder models dependencies and correlations between multi-granularity labels through a masked multi-head self-attention mechanism,and relates visual information to the semantic label information through a crossmodality attention mechanism.In this way,the proposed method preserves the relationships between labels at different granularity levels and takes into account the influence of different image regions on labels with different granularities.Evaluations on six public benchmarks qualitatively and quantitatively demonstrate the advantages of the proposed method.Our project is available at https://github.com/liuxindazz/mgs.

Image categorization with resource constraints： introduction, challenges and advances

As one of the most classic fields in computer vi- sion, image categorization has attracted widespread interests. Numerous algorithms have been proposed in the community, and many of them have advanced the state-of-the-art. How- ever, most existing algorithms are designed without consider- ation for the supply of computing resources. Therefore, when dealing with resource constrained tasks, these algorithms will fail to give satisfactory results. In this paper, we provide a comprehensive and in-depth introduction of recent develop- ments of the research in image categorization with resource constraints. While a large portion is based on our own work, we will also give a brief description of other elegant algo- rithms. Furthermore, we make an investigation into the re- cent developments of deep neural networks, with a focus on resource constrained deep nets.

Image categorization using a semantic hierarchy model with sparse set of salient regions

Image categorization in massive image database is an important problem. This paper proposes an approach for image categorization, using sparse set of salient semantic information and hierarchy semantic label tree （HSLT） model. First, to provide more critical image semantics, the proposed sparse set of salient regions only at the focuses of visual attention instead of the entire scene was formed by our proposed saliency detection model with incorporating low and high level feature and Shotton＇s semantic texton forests （STFs） method. Second, we also propose a new HSLT model in terms of the sparse regional semantic information to automatically build a semantic image hierarchy, which explicitly encodes a general to specific image relationship. And last, we archived image dataset using image hierarchical semantic, which is help to improve the performance of image organizing and browsing. Extension experimefital results showed that the use of semantic hierarchies as a hierarchical organizing frame- work provides a better image annotation and organization, improves the accuracy and reduces human＇s effort.

Facilitating fine-grained intra-urban dengue forecasting by integrating urban environments measured from street-view images

Background:Dengue fever(DF)is a mosquito-borne infectious disease that has threatened tropical and subtropical regions in recent decades.An early and targeted warning of a dengue epidemic is important for vector control.Current studies have primarily determined weather conditions to be the main factor for dengue forecasting,thereby neglecting that environmental suitability for mosquito breeding is also an important factor,especially in fine-grained intra-urban settings.Considering that street-view images are promising for depicting physical environments,this study proposes a framework for facilitating fine-grained intra-urban dengue forecasting by integrating the urban environments measured from street-view images.

细粒度图像分类综述

细粒度图像具有类内方差大、类间方差小的特点,致使细粒度图像分类(FGIC)的难度远高于传统的图像分类任务。介绍了FGIC的应用场景、任务难点、算法发展历程和相关的常用数据集,主要概述相关算法:基于局部检测的分类方法通常采用连接、求和及池化等操作,模型训练较为复杂,在实际应用中存在较多局限;基于线性特征的分类方法模仿人类视觉的两个神经通路分别进行识别和定位,分类效果相对较优;基于注意力机制的分类方法模拟人类观察外界事物的机制,先扫描全景,后锁定重点关注区域并形成注意力焦点,分类效果有进一步的提高。最后针对目前研究的不足,展望FGIC下一步的研究方向。

基于深度学习的食用菌分类研究

在介绍基于传统提取特征的食用菌分类方法基础上,通过利用卷积神经网络对食用菌进行深度分类的过程,阐述了基于深度学习的食用菌分类方法。试验数据证明深度学习方法在食用菌分类任务上取得了较高的准确率,明显优于传统的提取特征图像识别分类方法。

基于自适应三线性池化网络的细粒度图像分类

细粒度图像分类的关键在于提取图像中微妙的特征。现有基于弱监督方式的细粒度图像识别方法大多使用专家标注的边界注释辅助定位关键区域,存在标注成本高、训练过程复杂等问题。基于弱监督的双线性卷积神经网络方法因其学习到的特征空间更符合细粒度图像特性而具有一定的有效性,但忽略了层间的相互作用。针对细粒度图像识别领域存在的关键区域识别困难和层间交互关联弱的问题,融合二阶协方差通道注意力机制、自适应特征掩码与自适应三线性池化,提出自适应三线性池化网络ATP-Net,用于细粒度图像分类任务。通过二阶协方差通道注意力机制学习通道上的注意力向量,构建自适应特征掩码模块学习空间维上的注意力矩阵,设计自适应三线性池化模块学习特征的最终表示,以充分利用空间维、通道维上的信息。在CUB-200、Cars-196和Aircraft-1003个细粒度图像分类数据集上的实验结果表明,ATP-Net的分类精度分别为89.30%、94.20%和91.80%。

结合金字塔和长短期记忆网络的细粒度图像分类

细粒度图像分类任务的难点在于类间局部信息差异小.针对现有方法忽略低级特征的重要性,导致局部多样性缺失的问题,提出一种结合金字塔和长短期记忆网络的细粒度图像分类方法.首先,利用特征金字塔和挤压激励模块构建双向特征传递路径,以极少的参数量和计算量实现低级特征流动,从而提取局部的多级特征;接着,通过感兴趣区域引导的局部精炼金字塔,抑制显著区域,提高局部定位的多样性;最后在长短期记忆网络中引入注意力门控,调节各级特征中对细粒度信息的关注度,从而挖掘细粒度特征,并增强其鉴别性.在CUB-200-2011、Stanford Cars和FGVC-Aircraft数据集的分类准确率分别达到90.8%、95.9%和95.4%,明显优于目前主流的细粒度图像分类方法,相较于对比方法的最好结果分别提升1.2%、0.8%和2.0%.

基于深度表征学习的紫外极光卵图像聚类

极光受太阳风驱动的地磁亚暴等大尺度动力学影响,其形态及演化因不同的太阳风-磁层-电离层耦合作用可能表现不同。目前,极光卵及其形态的归类大多依据极光演化理论作主观定性分析,没有明确的分类标准,故难以借助统计分析方法和有监督分类模型开展客观定量研究。建立了基于深度表征学习的紫外极光卵图像聚类模型(MoCo-GMM),并利用空间环境参数设计了评估模型物理合理性的方法,在大规模POLAR卫星紫外极光卵图像数据上进行了实验,聚类结果不仅具有良好的簇内凝聚性和簇间分散性,且具备一定的物理可解释性,有效实现了基于图像的极光卵及其形态的客观归类。

基于无透镜成像的物联网气传病害早期预警系统

作物真菌性气传病害的特点意味着早期预警和干预能有效提高防治效果。现有孢子监测装置体积大、成本高、空间布局灵活性差,降低了病情评估的准确度。提出了一种基于无透镜成像的便携式孢子监测物联网系统。将空气中的孢子富集于载玻片上,采集24.396 mm2的无透镜成像,所设计的算法获得高质量重建结果,提取包括形态、幅相在内的多维度特征,实现孢子识别和分类。采用该方法对稻瘟病孢子进行富集和浓度检测,结果表明,该方法成像视场大、准确度高、检出限高,能在病害早期预警。其便携式、自动化特点有望为明确气传病害演化规律和预防流行传播提供数据支持。