Background:Brain functional connectivity under the naturalistic paradigm has been shown to be better at predicting individual behaviors than other brain states,such as rest and doing tasks.Nevertheless,the state-of-th...Background:Brain functional connectivity under the naturalistic paradigm has been shown to be better at predicting individual behaviors than other brain states,such as rest and doing tasks.Nevertheless,the state-of-the-art methods have found it difcult to achieve desirable results from movie watching paradigm functional magnetic resonance imaging(mfMRI)-induced brain functional connectivity,especially when there are fewer datasets.Incorporating other physical measurements into the prediction method may enhance accuracy.Eye tracking,becoming popular due to its portability and lower expense,can provide abundant behavioral features related to the output of human's cognition,and thus might supplement the mfMRI in observing participants'subconscious behaviors.However,there are very few studies on how to effectively integrate the multimodal information to strengthen the performance by a unified framework.objective:A fusion approach with mfMRI and eye tracking,based on convolution with edge-node switching in graph neural networks(CensNet),is proposed in this article.Methods:In this graph model,participants are designated as nodes,mfMRI derived functional connectivity as node features,and different eye-tracking features are used to compute similarity between participants to construct heterogeneous graph edges.By taking multiple graphs as different channels,we introduce squeeze-and-excitation attention module to CensNet(A-CensNet)to integrate graph embeddings from multiple channels into one.Results:The proposed model outperforms those using a single modality and single channel,and state-of-the-art methods.Conclusions:The results indicate that brain functional activities and eye behaviors might complement each other in interpreting trait-like phenotypes.展开更多
基金supported by the National Natural Science Foundation of China (31971288,U1801265,61936007,62276050,61976045,U20B2065,U1801265 and 61936007)the National Key R&D Pro-gram of China under Grant 2020AAA0105701+1 种基金High-level researcher start-up projects (Grant No.06100-22GH0202178)Innovation Foundation for Doctor Dissertation of Northwestern Poly-technical University CX2022052.
文摘Background:Brain functional connectivity under the naturalistic paradigm has been shown to be better at predicting individual behaviors than other brain states,such as rest and doing tasks.Nevertheless,the state-of-the-art methods have found it difcult to achieve desirable results from movie watching paradigm functional magnetic resonance imaging(mfMRI)-induced brain functional connectivity,especially when there are fewer datasets.Incorporating other physical measurements into the prediction method may enhance accuracy.Eye tracking,becoming popular due to its portability and lower expense,can provide abundant behavioral features related to the output of human's cognition,and thus might supplement the mfMRI in observing participants'subconscious behaviors.However,there are very few studies on how to effectively integrate the multimodal information to strengthen the performance by a unified framework.objective:A fusion approach with mfMRI and eye tracking,based on convolution with edge-node switching in graph neural networks(CensNet),is proposed in this article.Methods:In this graph model,participants are designated as nodes,mfMRI derived functional connectivity as node features,and different eye-tracking features are used to compute similarity between participants to construct heterogeneous graph edges.By taking multiple graphs as different channels,we introduce squeeze-and-excitation attention module to CensNet(A-CensNet)to integrate graph embeddings from multiple channels into one.Results:The proposed model outperforms those using a single modality and single channel,and state-of-the-art methods.Conclusions:The results indicate that brain functional activities and eye behaviors might complement each other in interpreting trait-like phenotypes.
