y-Tuning: an efficient tuning paradigm for large-scale pre-trained models via label representation learning

With current success of large-scale pre-trained models(PTMs),how efficiently adapting PTMs to downstream tasks has attracted tremendous attention,especially for PTMs with billions of parameters.Previous work focuses on designing parameter-efficient tuning paradigms but needs to save and compute the gradient of the whole computational graph.In this paper,we propose y-Tuning,an efficient yet effective paradigm to adapt frozen large-scale PTMs to specific downstream tasks.y-Tuning learns dense representations for labels y defined in a given task and aligns them to fixed feature representation.Without computing the gradients of text encoder at training phrase,y-Tuning is not only parameterefficient but also training-efficient.Experimental results show that for DeBERTaxxL with 1.6 billion parameters,y-Tuning achieves performance more than 96%of full fine-tuning on GLUE Benchmark with only 2%tunable parameters and much fewer training costs.

Classification of Conversational Sentences Using an Ensemble Pre-Trained Language Model with the Fine-Tuned Parameter

Sentence classification is the process of categorizing a sentence based on the context of the sentence.Sentence categorization requires more semantic highlights than other tasks,such as dependence parsing,which requires more syntactic elements.Most existing strategies focus on the general semantics of a conversation without involving the context of the sentence,recognizing the progress and comparing impacts.An ensemble pre-trained language model was taken up here to classify the conversation sentences from the conversation corpus.The conversational sentences are classified into four categories:information,question,directive,and commission.These classification label sequences are for analyzing the conversation progress and predicting the pecking order of the conversation.Ensemble of Bidirectional Encoder for Representation of Transformer(BERT),Robustly Optimized BERT pretraining Approach(RoBERTa),Generative Pre-Trained Transformer(GPT),DistilBERT and Generalized Autoregressive Pretraining for Language Understanding(XLNet)models are trained on conversation corpus with hyperparameters.Hyperparameter tuning approach is carried out for better performance on sentence classification.This Ensemble of Pre-trained Language Models with a Hyperparameter Tuning(EPLM-HT)system is trained on an annotated conversation dataset.The proposed approach outperformed compared to the base BERT,GPT,DistilBERT and XLNet transformer models.The proposed ensemble model with the fine-tuned parameters achieved an F1_score of 0.88.

The enlightenment of artificial intelligence large-scale model on the research of intelligent eye diagnosis in traditional Chinese medicine

Eye diagnosis is a method for inspecting systemic diseases and syndromes by observing the eyes.With the development of intelligent diagnosis in traditional Chinese medicine(TCM);artificial intelligence(AI)can improve the accuracy and efficiency of eye diagnosis.However;the research on intelligent eye diagnosis still faces many challenges;including the lack of standardized and precisely labeled data;multi-modal information analysis;and artificial in-telligence models for syndrome differentiation.The widespread application of AI models in medicine provides new insights and opportunities for the research of eye diagnosis intelli-gence.This study elaborates on the three key technologies of AI models in the intelligent ap-plication of TCM eye diagnosis;and explores the implications for the research of eye diagno-sis intelligence.First;a database concerning eye diagnosis was established based on self-su-pervised learning so as to solve the issues related to the lack of standardized and precisely la-beled data.Next;the cross-modal understanding and generation of deep neural network models to address the problem of lacking multi-modal information analysis.Last;the build-ing of data-driven models for eye diagnosis to tackle the issue of the absence of syndrome dif-ferentiation models.In summary;research on intelligent eye diagnosis has great potential to be applied the surge of AI model applications.

Adapter Based on Pre-Trained Language Models for Classification of Medical Text

We present an approach to classify medical text at a sentence level automatically.Given the inherent complexity of medical text classification,we employ adapters based on pre-trained language models to extract information from medical text,facilitating more accurate classification while minimizing the number of trainable parameters.Extensive experiments conducted on various datasets demonstrate the effectiveness of our approach.

Modular Extremely Large-Scale Array Communication:Near-Field Modelling and Performance Analysis

This paper investigates the wireless communication with a novel architecture of antenna arrays,termed modular extremely large-scale array(XLarray),where array elements of an extremely large number/size are regularly mounted on a shared platform with both horizontally and vertically interlaced modules.Each module consists of a moderate/flexible number of array elements with the inter-element distance typically in the order of the signal wavelength,while different modules are separated by the relatively large inter-module distance for convenience of practical deployment.By accurately modelling the signal amplitudes and phases,as well as projected apertures across all modular elements,we analyse the near-field signal-to-noise ratio(SNR)performance for modular XL-array communications.Based on the non-uniform spherical wave(NUSW)modelling,the closed-form SNR expression is derived in terms of key system parameters,such as the overall modular array size,distances of adjacent modules along all dimensions,and the user's three-dimensional(3D)location.In addition,with the number of modules in different dimensions increasing infinitely,the asymptotic SNR scaling laws are revealed.Furthermore,we show that our proposed near-field modelling and performance analysis include the results for existing array architectures/modelling as special cases,e.g.,the collocated XL-array architecture,the uniform plane wave(UPW)based far-field modelling,and the modular extremely large-scale uniform linear array(XL-ULA)of onedimension.Extensive simulation results are presented to validate our findings.

Comparison of Pile-Soil-Structure Interaction Modeling Techniques for A 10-MW Large-Scale Monopile Wind Turbine Model Under Wind and Wave Conditions

Considering the large diameter effect of piles,the influence of different pile-soil analysis methods on the design of monopile foundations for offshore wind turbines has become an urgent problem to be solved.Three different pile-soil models were used to study a large 10 MW monopile wind turbine.By modeling the three models in the SACS software,this paper analyzed the motion response of the overall structure under the conditions of wind and waves.According to the given working conditions,this paper concludes that under the condition of independent wind,the average value of the tower top x-displacement of the rigid connection method is the smalle st,and the standard deviation is the smallest under the condition of independent wave.The results obtained by the p-y curve method are the most conservative.

Construction and application of knowledge graph for grid dispatch fault handling based on pre-trained model

With the construction of new power systems,the power grid has become extremely large,with an increasing proportion of new energy and AC/DC hybrid connections.The dynamic characteristics and fault patterns of the power grid are complex;additionally,power grid control is difficult,operation risks are high,and the task of fault handling is arduous.Traditional power-grid fault handling relies primarily on human experience.The difference in and lack of knowledge reserve of control personnel restrict the accuracy and timeliness of fault handling.Therefore,this mode of operation is no longer suitable for the requirements of new systems.Based on the multi-source heterogeneous data of power grid dispatch,this paper proposes a joint entity–relationship extraction method for power-grid dispatch fault processing based on a pre-trained model,constructs a knowledge graph of power-grid dispatch fault processing and designs,and develops a fault-processing auxiliary decision-making system based on the knowledge graph.It was applied to study a provincial dispatch control center,and it effectively improved the accident processing ability and intelligent level of accident management and control of the power grid.

Leveraging Vision-Language Pre-Trained Model and Contrastive Learning for Enhanced Multimodal Sentiment Analysis

Multimodal sentiment analysis is an essential area of research in artificial intelligence that combines multiple modes,such as text and image,to accurately assess sentiment.However,conventional approaches that rely on unimodal pre-trained models for feature extraction from each modality often overlook the intrinsic connections of semantic information between modalities.This limitation is attributed to their training on unimodal data,and necessitates the use of complex fusion mechanisms for sentiment analysis.In this study,we present a novel approach that combines a vision-language pre-trained model with a proposed multimodal contrastive learning method.Our approach harnesses the power of transfer learning by utilizing a vision-language pre-trained model to extract both visual and textual representations in a unified framework.We employ a Transformer architecture to integrate these representations,thereby enabling the capture of rich semantic infor-mation in image-text pairs.To further enhance the representation learning of these pairs,we introduce our proposed multimodal contrastive learning method,which leads to improved performance in sentiment analysis tasks.Our approach is evaluated through extensive experiments on two publicly accessible datasets,where we demonstrate its effectiveness.We achieve a significant improvement in sentiment analysis accuracy,indicating the supe-riority of our approach over existing techniques.These results highlight the potential of multimodal sentiment analysis and underscore the importance of considering the intrinsic semantic connections between modalities for accurate sentiment assessment.

Construction of Risk Assessment Application Model of Epidemic Disease in Large-scale Pig Farms

The application model of epidemic disease assessment technology for Web-based large-scale pig farm was expounded from the identification of epidemic disease risk factors, construction of risk assessment model and development of risk assessment system. The assessed pig farm uploaded the epidemic disease risk data information through on-line answering evaluating questionnaire to get the immediate evaluation report. The model could enhance the risk communication between pig farm veterinarian, manager and veterinary experts to help farm system understand and find disease risk factors, assess and report the potential high risk items of the pig farm in the three systems of engineering epidemic disease prevention technology, biological safety and immune monitoring, and promote the improvement and perfection of epidemic disease prevention and control measures.

A Classification–Detection Approach of COVID-19 Based on Chest X-ray and CT by Using Keras Pre-Trained Deep Learning Models

The Coronavirus Disease 2019(COVID-19)is wreaking havoc around the world,bring out that the enormous pressure on national health and medical staff systems.One of the most effective and critical steps in the fight against COVID-19,is to examine the patient’s lungs based on the Chest X-ray and CT generated by radiation imaging.In this paper,five keras-related deep learning models:ResNet50,InceptionResNetV2,Xception,transfer learning and pre-trained VGGNet16 is applied to formulate an classification-detection approaches of COVID-19.Two benchmark methods SVM(Support Vector Machine),CNN(Conventional Neural Networks)are provided to compare with the classification-detection approaches based on the performance indicators,i.e.,precision,recall,F1 scores,confusion matrix,classification accuracy and three types of AUC(Area Under Curve).The highest classification accuracy derived by classification-detection based on 5857 Chest X-rays and 767 Chest CTs are respectively 84%and 75%,which shows that the keras-related deep learning approaches facilitate accurate and effective COVID-19-assisted detection.

Long Short-Term Memory Recurrent Neural Network-Based Acoustic Model Using Connectionist Temporal Classification on a Large-Scale Training Corpus

A Long Short-Term Memory(LSTM) Recurrent Neural Network(RNN) has driven tremendous improvements on an acoustic model based on Gaussian Mixture Model(GMM). However, these models based on a hybrid method require a forced aligned Hidden Markov Model(HMM) state sequence obtained from the GMM-based acoustic model. Therefore, it requires a long computation time for training both the GMM-based acoustic model and a deep learning-based acoustic model. In order to solve this problem, an acoustic model using CTC algorithm is proposed. CTC algorithm does not require the GMM-based acoustic model because it does not use the forced aligned HMM state sequence. However, previous works on a LSTM RNN-based acoustic model using CTC used a small-scale training corpus. In this paper, the LSTM RNN-based acoustic model using CTC is trained on a large-scale training corpus and its performance is evaluated. The implemented acoustic model has a performance of 6.18% and 15.01% in terms of Word Error Rate(WER) for clean speech and noisy speech, respectively. This is similar to a performance of the acoustic model based on the hybrid method.

Validating the Runoff from the PRECIS Model Using a Large-Scale Routing Model

The streamflow over the Yellow River basin is simulated using the PRECIS （Providing REgional Climates for Impacts Studies） regional climate model driven by 15-year （1979-1993） ECMWF reanalysis data as the initial and lateral boundary conditions and an off-line large-scale routing model （LRM）. The LRM uses physical catchment and river channel information and allows streamflow to be predicted for large continental rivers with a 1°×1° spatial resolution. The results show that the PRECIS model can reproduce the general southeast to northwest gradient distribution of the precipitation over the Yellow River basin, The PRECIS- LRM model combination has the capability to simulate the seasonal and annual streamflow over the Yellow River basin. The simulated streamflow is generally coincident with the naturalized streamflow both in timing and in magnitude.

Research status and application of artificial intelligence large models in the oil and gas industry

This article elucidates the concept of large model technology,summarizes the research status of large model technology both domestically and internationally,provides an overview of the application status of large models in vertical industries,outlines the challenges and issues confronted in applying large models in the oil and gas sector,and offers prospects for the application of large models in the oil and gas industry.The existing large models can be briefly divided into three categories:large language models,visual large models,and multimodal large models.The application of large models in the oil and gas industry is still in its infancy.Based on open-source large language models,some oil and gas enterprises have released large language model products using methods like fine-tuning and retrieval augmented generation.Scholars have attempted to develop scenario-specific models for oil and gas operations by using visual/multimodal foundation models.A few researchers have constructed pre-trained foundation models for seismic data processing and interpretation,as well as core analysis.The application of large models in the oil and gas industry faces challenges such as current data quantity and quality being difficult to support the training of large models,high research and development costs,and poor algorithm autonomy and control.The application of large models should be guided by the needs of oil and gas business,taking the application of large models as an opportunity to improve data lifecycle management,enhance data governance capabilities,promote the construction of computing power,strengthen the construction of“artificial intelligence+energy”composite teams,and boost the autonomy and control of large model technology.

Improved Diurnal Cycle of Precipitation on Land in a Global Non-Hydrostatic Model Using a Revised NSAS Deep Convective Scheme

In relatively coarse-resolution atmospheric models,cumulus parameterization helps account for the effect of subgridscale convection,which produces supplemental rainfall to the grid-scale precipitation and impacts the diurnal cycle of precipitation.In this study,the diurnal cycle of precipitation was studied using the new simplified Arakawa-Schubert scheme in a global non-hydrostatic atmospheric model,i.e.,the Yin-Yang-grid Unified Model for the Atmosphere.Two new diagnostic closures and a convective trigger function were suggested to emphasize the job of the cloud work function corresponding to the free tropospheric large-scale forcing.Numerical results of the 0.25-degree model in 3-month batched real-case simulations revealed an improvement in the diurnal precipitation variation by using a revised trigger function with an enhanced dynamical constraint on the convective initiation and a suitable threshold of the trigger.By reducing the occurrence of convection during peak solar radiation hours,the revised scheme was shown to be effective in delaying the appearance of early-afternoon rainfall peaks over most land areas and accentuating the nocturnal peaks that were wrongly concealed by the more substantial afternoon peak.In addition,the revised scheme enhanced the simulation capability of the precipitation probability density function,such as increasing the extremely low-and high-intensity precipitation events and decreasing small and moderate rainfall events,which contributed to the reduction of precipitation bias over mid-latitude and tropical land areas.

Evaluating the role of large language models in inflammatory bowel disease patient information

This letter evaluates the article by Gravina et al on ChatGPT’s potential in providing medical information for inflammatory bowel disease patients.While promising,it highlights the need for advanced techniques like reasoning+action and retrieval-augmented generation to improve accuracy and reliability.Emphasizing that simple question and answer testing is insufficient,it calls for more nuanced evaluation methods to truly gauge large language models’capabilities in clinical applications.

Advances of Model Order Reduction Research in Large-scale System Simulation

Model Order Reduction (MOR) plays more and more imp or tant role in complex system simulation, design and control recently. For example , for the large-size space structures, VLSI and MEMS (Micro-ElectroMechanical Systems) etc., in order to shorten the development cost, increase the system co ntrolling accuracy and reduce the complexity of controllers, the reduced order model must be constructed. Even in Virtual Reality (VR), the simulation and d isplay must be in real-time, the model order must be reduced too. The recent advances of MOR research are overviewed in the article. The MOR theor y and methods may be classified as Singular Value decomposition (SVD) based, the Krylov subspace based and others. The merits and demerits of the different meth ods are analyzed, and the existed problems are pointed out. Moreover, the applic ation’s fields are overviewed, and the potential applications are forecaste d. After the existed problems analyzed, the future work is described. There are som e problems in the traditional methods such as SVD and Krylov subspace, they are that it’s difficult to (1)guarantee the stability of the original system, (2) b e adaptive to nonlinear system, and (3) control the modeling accuracy. The f uture works may be solving the above problems on the foundation of the tradition al methods, and applying other methods such as wavelet or signal compression.

Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey

With the urgent demand for generalized deep models,many pre-trained big models are proposed,such as bidirectional encoder representations(BERT),vision transformer(ViT),generative pre-trained transformers(GPT),etc.Inspired by the success of these models in single domains(like computer vision and natural language processing),the multi-modal pre-trained big models have also drawn more and more attention in recent years.In this work,we give a comprehensive survey of these models and hope this paper could provide new insights and helps fresh researchers to track the most cutting-edge works.Specifically,we firstly introduce the background of multi-modal pre-training by reviewing the conventional deep learning,pre-training works in natural language process,computer vision,and speech.Then,we introduce the task definition,key challenges,and advantages of multi-modal pre-training models(MM-PTMs),and discuss the MM-PTMs with a focus on data,objectives,network architectures,and knowledge enhanced pre-training.After that,we introduce the downstream tasks used for the validation of large-scale MM-PTMs,including generative,classification,and regression tasks.We also give visualization and analysis of the model parameters and results on representative downstream tasks.Finally,we point out possible research directions for this topic that may benefit future works.In addition,we maintain a continuously updated paper list for large-scale pre-trained multi-modal big models:https://github.com/wangxiao5791509/MultiModal_BigModels_Survey.

Porcine Breeding Management in a Large-scale Piggery with Microbial Fermentation Bed

[Objective] The behavior of eating, drinking, defecating and peeing of 1 500 pigs in a large-scale microbial fermentation bed-equipped piggery was observed. We hoped to find some simple indicators that could reflect the health status of swinery and to provide experience for the swinery performance management in large-scale microbial fermentation bed-equipped piggery. [Method] The body weight （BW）, daily BW gain, feed intake and other indicators of different-day-old pigs were recorded in details. Based on the recorded data, the models between BW, BW gain, average daily feed intake and feed/gain ratio and growth days （d） were established. In addition, the incidences of pox-like macula （dermatitis）, diarrhea （gastrointestinal disease）, cough （respiratory disease）, stiff pig （malnutrition）, conjunctivitis （eye disease） and foot inflection （trauma） among fattening pigs were also investigated. [Result] The BW range, average BW, daily BW gain, breeding days, daily feed intake range, average daily feed intake, staged feed intake, accumulated feed intake, feed/gain ratio and accumulated feed/gain ratio of different-day-old pigs were studied, respectively. Four dynamic models were established for the growth of pigs： （1） the BW （y）-age （x） mod- el： y=0.758 9x-19.883 （3=0.993 7）; （2） the BW gain （y）-age （x） model： y=1.039 5x05051 （F=0.885 4）; （3） the average daily feed intake （y）-age （x） model： y=0.023 5x-0.334 3 （F=0.991 7）; （4） the feed/gain ratio （y）-age （x） model： y=0.022x＋0.427 8 （P=0.988 5）. Based on these models, the corresponding theoretical growth value of pigs at different growth stage could be predicted. The main diseases occurred among the swinery in the large-scale microbial fermentation bed piggery included pox-like macula （dermatitis）, diarrhea （gastrointestinal disease）, cough （respiratory disease）, stiff pig （mal- nutrition）, conjunctivitis （eye disease） and foot inflection （trauma）. The deadly infec- tious diseases had been not found among the pigs. [Conclusion] When the actual BW, BW gain, average daily feed intake and feed/gain ratio were all lower than the theoretical values predicted by the models, the management should be enhanced. The average daily feed intake of 60 to 65-day-old pigs was lower than the theoretic value, indicating that the pigs could not adapt nicely to the fermentation bed at the very early stage. When the pigs grew up to 70 to 75 d old, the average daily feed intake was higher than the theoretical value, indicating that the pigs had adapted to the fermentation bed. In particularly, average daily feed intake of 75-day-old pigs was higher than the theoretical value by 21%. It was suggested the fermentation bed was conducive to the growth of pigs. Considering the occurrence of diseases among pigs, the overall incidence was relatively low. The incidence of each disease was all lower than 10% with little difficulty in treating. If the management of mattress was strength- ened, such as paying attention to feeding and keeping water clean, many diseases could heal by themselves.

A Game-Theoretic Perspective on Resource Management for Large-Scale UAV Communication Networks

As a result of rapid development in electronics and communication technology,large-scale unmanned aerial vehicles(UAVs)are harnessed for various promising applications in a coordinated manner.Although it poses numerous advantages,resource management among various domains in large-scale UAV communication networks is the key challenge to be solved urgently.Specifically,due to the inherent requirements and future development trend,distributed resource management is suitable.In this article,we investigate the resource management problem for large-scale UAV communication networks from game-theoretic perspective which are exactly coincident with the distributed and autonomous manner.By exploring the inherent features,the distinctive challenges are discussed.Then,we explore several gametheoretic models that not only combat the challenges but also have broad application prospects.We provide the basics of each game-theoretic model and discuss the potential applications for resource management in large-scale UAV communication networks.Specifically,mean-field game,graphical game,Stackelberg game,coalition game and potential game are included.After that,we propose two innovative case studies to highlight the feasibility of such novel game-theoretic models.Finally,we give some future research directions to shed light on future opportunities and applications.

Temperature control for thermal treatment of aluminum alloy in a large-scale vertical quench furnace

The temperature control of the large-scale vertical quench furnace is very difficult due to its huge volume and complex thermal exchanges. To meet the technical requirement of the quenching process, a temperature control system which integrates temperature calibration and temperature uniformity control is developed for the thermal treatment of aluminum alloy workpieces in the large-scale vertical quench furnace. To obtain the aluminum alloy workpiece temperature, an air heat transfer model is newly established to describe the temperature gradient distribution so that the immeasurable workpiece temperature can be calibrated from the available thermocouple temperature. To satisfy the uniformity control of the furnace temperature, a second order partial differential equation(PDE) is derived to describe the thermal dynamics inside the vertical quench furnace. Based on the PDE, a decoupling matrix is constructed to solve the coupling issue and decouple the heating process into multiple independent heating subsystems. Then, using the expert control rule to find a compromise of temperature rising time and overshoot during the quenching process. The developed temperature control system has been successfully applied to a 31 m large-scale vertical quench furnace, and the industrial running results show the significant improvement of the temperature uniformity, lower overshoot and shortened processing time.