基于流模型的荧光图像超分辨率

Flow⁃Based Model for Fluorescence Image Super⁃Resolution

荧光图像超分辨率方法可以将低分辨率荧光图像重建为超分辨率图像。近几年来,基于深度学习的方法通过学习配对的高/低分辨率图像间的映射函数,取得了较好性能。然而,一幅低分辨率荧光图像可能由不同的超分辨图像退化而来,这使得荧光图像超分辨率任务属于欠定问题。现有的深度学习方法通常学习一个确定性的映射函数,而忽略了超分辨率任务的欠定特征。针对这一问题,提出了一种基于流模型的荧光图像超分辨率方法。该方法在给定低分辨率图像的条件下,可以重建出一系列超分辨率图像的分布,更符合超分辨率任务的欠定性质。此外,进一步提出了频域-空域联合注意力模块,用于提取荧光图像的潜在特征,使网络能够充分利用数据在时间和空间上的先验分布,从而获得更准确的图像重建结果。实验结果表明,此方法重建图像的感知指标和重建效果均有所提升。

Objective Existing deep learning-based methods for fluorescence image super-resolution can be broadly classified into two categories:those guided by peak signal-to-noise ratio(PSNR)and those guided by perceptual considerations.The former tends to produce excessively smoothed prediction results while the latter mitigates the over smoothing issue considerably;however,both categories overlook the ill-posed nature of the super-resolution task.This study proposes a fluorescence image super-resolution method based on flow models capable of reconstructing multiple realistic super-resolution images that align with the ill-posed nature of super-resolution tasks.Moreover,microscopy imaging is conducted in continuous time sequences naturally containing temporal information.However,current methods often focus solely on individual image frames for super-resolution reconstruction,completely disregarding the temporal information between adjacent frames.Additionally,structures in similar biological samples exhibit a certain degree of similarity,and fluorescence images collected possess internal self-similarity in the spatial domain.To fully leverage the temporal and spatial information present in fluorescence images,this study proposes a frequency-and spatial-domain joint attention module.This module aims to focus on features that significantly contribute to the prediction results,obtaining more accurate reconstruction outcomes.Similar to most supervised learning methods,our approach has a limitation in that it requires labeled paired image sets for training the network model.Generalization performance may significantly decline when applying the model to a test set with a distribution different from the training set.Acquiring labeled paired training data is not always feasible in practical applications.Therefore,future work may need to address the challenge of cross-dataset super-resolution reconstruction,considering optimization strategies and network improvements from a domain adaptation perspective.Methods This study introduces a flow-model-based multi-frame dual-domain attention flow network.Given a low-resolution image,the network learns the distribution of super-resolution images using flow models,enabling the reconstruction of multiple realistic super-resolution images to address the underdetermined nature of super-resolution tasks.Additionally,as the imaging process is typically continuous,the acquired raw data from a microscope has temporal relationships with adjacent frames.However,existing deep learning-based fluorescence image super-resolution methods often neglect the temporal priors present in multiple input frames over time.Moreover,biological sample structures exhibit internal self-similarity.Therefore,by constructing a frequency-domain and spatial-domain joint attention module,the network is guided to focus extensively on features that contribute significantly to the prediction results,further enhancing the network’s performance.The proposed flow-model-based multi-frame dual-domain attention flow network consists of a flow model and a frequency-domain and spatial-domain joint attention module.The flow model,composed of multiple reversible modules,facilitates a reversible mapping between the target and latent space distribution.The frequency-domain and spatial-domain joint attention module achieves conditional feature extraction and includes a set of frequency-and spatial-domain attention blocks.These blocks comprise Fourier channel attention blocks,spatial attention blocks,and convolutional layers,serving to extract temporal,spatial,and aggregated features from the fluorescence image,respectively.Furthermore,the study employs skip connections to enable feature reuse and prevent gradient vanishing.Results and Discussions This study demonstrates the importance of temporal information by comparing the proposed method,a multi-frame dual-domain attention flow network(MDAFN),with a single-frame dual-domain attention flow network(SDAFN).Quantitative evaluation metrics include PSNR and learned perceptual image patch similarity(LPIPS).Experimental results indicate that the MDAFN outperforms the SDAFN,and the indexes of PSNR and LPIPS on the three data sets are shown in Table 1.Moreover,visually,the images reconstructed using the MDAFN exhibit improvement over those generated using the SDAFN(Figs.7‒9).Finally,a comparison between the proposed method and state-of-the-art super-resolution reconstruction methods is presented.The results indicate that when the standard deviation of hyperparameters is set to zero,the PSNR of the super-resolved images obtained using the proposed method is comparable or even superior to that obtained using other methods.For LPIPS,the proposed method outperforms other methods.When the standard deviation is greater than zero,the LPIPS obtained using the proposed method is further decreased across the three datasets(Table 2).The reconstructed results using the proposed method visually resemble the ground-truth images more closely.In contrast,other methods generate over-smoothed,signal-depleted,or artificially enhanced reconstructions with poorer subjective quality(Figs.10‒13).Conclusions This study proposes an MDAFN for high-quality super-resolution reconstruction of fluorescence images.Unlike conventional neural networks that directly learn deterministic mapping functions,our approach can predict various possible super-resolution images for a given low-resolution wide-field image,addressing the underdetermined nature of the one-to-many relationship in super-resolution tasks.Additionally,considering the high internal self-similarity of structures in similar live cell samples in both temporal and spatial dimensions of fluorescence images,we further introduce a frequency-and spatial-domain joint attention module based on multi-temporal input.This module aims to focus more on features contributing significantly to the prediction results,yielding more accurate predictions.Experimental results demonstrate that the proposed method outperforms other methods in terms of super-resolution image quality.