Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey

With the urgent demand for generalized deep models,many pre-trained big models are proposed,such as bidirectional encoder representations(BERT),vision transformer(ViT),generative pre-trained transformers(GPT),etc.Inspired by the success of these models in single domains(like computer vision and natural language processing),the multi-modal pre-trained big models have also drawn more and more attention in recent years.In this work,we give a comprehensive survey of these models and hope this paper could provide new insights and helps fresh researchers to track the most cutting-edge works.Specifically,we firstly introduce the background of multi-modal pre-training by reviewing the conventional deep learning,pre-training works in natural language process,computer vision,and speech.Then,we introduce the task definition,key challenges,and advantages of multi-modal pre-training models(MM-PTMs),and discuss the MM-PTMs with a focus on data,objectives,network architectures,and knowledge enhanced pre-training.After that,we introduce the downstream tasks used for the validation of large-scale MM-PTMs,including generative,classification,and regression tasks.We also give visualization and analysis of the model parameters and results on representative downstream tasks.Finally,we point out possible research directions for this topic that may benefit future works.In addition,we maintain a continuously updated paper list for large-scale pre-trained multi-modal big models:https://github.com/wangxiao5791509/MultiModal_BigModels_Survey.

Improved energy storage performance of bismuth sodium titanate-based lead-free relaxor ferroelectric ceramics via Bi-containing complex ions doping

Lead-free dielectric ceramics can be used to make quick charge-discharge capacitor devices due to their high power density.Their use in advanced electronic systems,however,has been hampered by their poor energy storage performance(ESP),which includes low energy storage efficiency and recoverable energy storage density(Wrec).In this work,we adopted a combinatorial optimization strategy to improve the ESP in(Bi_(0.5)Na_(0.5))TiO_(3)(BNT)-based relaxor ferroelectric ceramics.To begin,the Bi-containing complex ions Bi(Mg_(2/3)Nb_(1/3))O_(3)(BMN)were introduced into a BNT-based matrix in order to improve the diffuse phase transition,increase Bi-O bond coupling,avoid macro domain development,and limit polarization response hysteresis.Second,the viscous polymer process was employed to reduce sample thickness and porosity,resulting in an apparent increase in breakdown strength in(1-x)[0.7(Bi_(1/2)Na_(1/2))TiO_(3)]-0.3SrTiO_(3)-xBi(Mg_(2/3)Nb_(1/3))O_(3)(BS-xBMN)ceramics.Finally,in x=0.20 composition,an amazing Wrecof 5.62 J·cm^(-3)and an ultra-high efficiency of 91.4%were simultaneously achieved at a relatively low field of 330 kV·cm^(-1),together with remarkable temperature stability in the temperature range of 30-140℃(3.5 J·cm^(-3)±5%variation).This research presents a new lead-free dielectric material with superior ESP for use in pulsed power capacitors.