Advances of Pipeline Model Parallelism for Deep Learning Training:An Overview

Deep learning has become the cornerstone of artificial intelligence,playing an increasingly important role in human production and lifestyle.However,as the complexity of problem-solving increases,deep learning models become increasingly intricate,resulting in a proliferation of large language models with an astonishing number of parameters.Pipeline model parallelism(PMP)has emerged as one of the mainstream approaches to addressing the significant challenge of training“big models”.This paper presents a comprehensive review of PMP.It covers the basic concepts and main challenges of PMP.It also comprehensively compares synchronous and asynchronous pipeline schedules for PMP approaches,and discusses the main techniques to achieve load balance for both intra-node and inter-node training.Furthermore,the main techniques to optimize computation,storage,and communication are presented,with potential research directions being discussed.

Optimization Techniques for GPU-Based Parallel Programming Models in High-Performance

This study embarks on a comprehensive examination of optimization techniques within GPU-based parallel programming models,pivotal for advancing high-performance computing(HPC).Emphasizing the transition of GPUs from graphic-centric processors to versatile computing units,it delves into the nuanced optimization of memory access,thread management,algorithmic design,and data structures.These optimizations are critical for exploiting the parallel processing capabilities of GPUs,addressingboth the theoretical frameworks and practical implementations.By integrating advanced strategies such as memory coalescing,dynamic scheduling,and parallel algorithmic transformations,this research aims to significantly elevate computational efficiency and throughput.The findings underscore the potential of optimized GPU programming to revolutionize computational tasks across various domains,highlighting a pathway towards achieving unparalleled processing power and efficiency in HPC environments.The paper not only contributes to the academic discourse on GPU optimization but also provides actionable insights for developers,fostering advancements in computational sciences and technology.

Attenuate Class Imbalance Problem for Pneumonia Diagnosis Using Ensemble Parallel Stacked Pre-Trained Models

Pneumonia is an acute lung infection that has caused many fatalitiesglobally. Radiologists often employ chest X-rays to identify pneumoniasince they are presently the most effective imaging method for this purpose.Computer-aided diagnosis of pneumonia using deep learning techniques iswidely used due to its effectiveness and performance. In the proposed method,the Synthetic Minority Oversampling Technique (SMOTE) approach is usedto eliminate the class imbalance in the X-ray dataset. To compensate forthe paucity of accessible data, pre-trained transfer learning is used, and anensemble Convolutional Neural Network (CNN) model is developed. Theensemble model consists of all possible combinations of the MobileNetv2,Visual Geometry Group (VGG16), and DenseNet169 models. MobileNetV2and DenseNet169 performed well in the Single classifier model, with anaccuracy of 94%, while the ensemble model (MobileNetV2+DenseNet169)achieved an accuracy of 96.9%. Using the data synchronous parallel modelin Distributed Tensorflow, the training process accelerated performance by98.6% and outperformed other conventional approaches.

Numerical and experimental investigation on hydraulic-electric rock fragmentation of heterogeneous granite

Hydraulic-electric rock fragmentation(HERF)plays a significant role in improving the efficiency of high voltage pulse rock breaking.However,the underlying mechanism of HERF remains unclear.In this study,considering the heterogeneity of the rock,microscopic thermodynamic properties,and shockwave time domain waveforms,based on the shockwave model,digital imaging technology and the discrete element method,the cyclic loading numerical simulations of HERF is achieved by coupling electrical,thermal,and solid mechanics under different formation temperatures,confining pressure,initial peak voltage,electrode bit diameter,and loading times.Meanwhile,the HERF discharge system is conducive to the laboratory experiments with various electrical parameters and the resulting broken pits are numerically reconstructed to obtain the geometric parameters.The results show that,the completely broken area consists of powdery rock debris.In the pre-broken zone,the mineral cementation of the rock determines the transition of type CⅠcracks to type CⅡand type CⅢcracks.Furthermore,the peak pressure of the shockwave increased with initial peak voltage but decreased with electrode bit diameter,while the wave front time reduced.Moreover,increasing well depth,formation temperature and confining pressure augment and inhibit HERF,but once confining pressure surpassed the threshold of 60 MPa for 152.40,215.90,and 228.60 mm electrode bits,and 40 MPa for 309.88 mm electrode bits,HERF is promoted.Additionally,for the same kind of rock,the volume and width of the broken pit increase with higher initial peak voltage and rock fissures will promote HERF.Eventually,the electrode drill bit with a 215.90 mm diameter is more suitable for drilling pink granite.This research contributes to a better microscopic understanding of HERF and provides valuable insights for electrode bit selection,as well as the optimization of circuit parameters for HERF technology.

A Novel Interconnect Crosstalk Parallel RLC Analyzable Model Based on the 65nm CMOS Process

Based on the 65nm CMOS process,a novel parallel RLC coupling interconnect analytical model is presented synthetically considering parasitical capacitive and parasitical inductive effects. Applying function approximation and model order-reduction to the model, we derive a closed-form and time-domain waveform for the far-end crosstalk of a victim line under ramp input transition. For various interconnect coupling sizes, the proposed RLC coupling analytical model enables the estimation of the crosstalk voltage within 2.50% error compared with Hspice simulation in a 65nm CMOS process. This model can be used in computer-aided-design of nanometer SOCs.

Heterogeneities of grain boundary contact for simulation of laboratoryscale mechanical behavior of granitic rocks

From a practical point of view,grain structure heterogeneities are key parameters that control the rock response and still remains a challenge to incorporate in a quantitative manner.One of the less discussed topics in the context of the grain-based model(GBM)in the particle flow code(PFC)is the contact heterogeneities and the appropriate contact model to mimic the grain boundary behavior.Generally,the smooth joint(SJ)model and linear parallel bond(LPB)model are used to simulate the grain boundary behavior.However,the literature does not document the suitability of different models for specific problems.Another challenge in implementing GBM in PFC is that only a single bonding parameter is used at the grain boundaries.The aim of this study is to investigate the responses of a laboratory-scale specimen with SJ and LPB models,considering grain boundary heterogeneous and homogeneous contact parameters.Uniaxial and biaxial compression tests are performed to calibrate the response of Creighton granite.The stressestrain curves,volumetric dilation,inter-crack(crack in the grain boundary),and intra-crack(crack within the grain)development,and failure patterns associated with different contact models are examined.It was found that both the SJ and LPB models can reproduce the pre-peak behavior observed for a granitic rock type.However,the LPB model is unable to reproduce the post-peak behavior.Due to the large interlocking effect originating from the balls in contact and the ball size in the LPB model,local dilation is induced at the grain boundaries.This overestimates the volumetric dilation and residual shear strength.The LPB model tends to result in discontinuous inter-cracks and stress localization in the rock specimen,resulting in fine fragments at the rock surface during failure.

Influence of heterogeneity on rock strength and stiffness using discrete element method and parallel bond model

The particulate discrete element method(DEM) can be employed to capture the response of rock,provided that appropriate bonding models are used to cement the particles to each other.Simulations of laboratory tests are important to establish the extent to which those models can capture realistic rock behaviors.Hitherto the focus in such comparison studies has either been on homogeneous specimens or use of two-dimensional(2D) models.In situ rock formations are often heterogeneous,thus exploring the ability of this type of models to capture heterogeneous material behavior is important to facilitate their use in design analysis.In situ stress states are basically three-dimensional(3D),and therefore it is important to develop 3D models for this purpose.This paper revisits an earlier experimental study on heterogeneous specimens,of which the relative proportions of weaker material(siltstone) and stronger,harder material(sandstone) were varied in a controlled manner.Using a 3D DEM model with the parallel bond model,virtual heterogeneous specimens were created.The overall responses in terms of variations in strength and stiffness with different percentages of weaker material(siltstone) were shown to agree with the experimental observations.There was also a good qualitative agreement in the failure patterns observed in the experiments and the simulations,suggesting that the DEM data enabled analysis of the initiation of localizations and micro fractures in the specimens.

Parallel Computation of Shallow-water Model on Workstations Cluster

ＰａｒａｌｌｅｌＣｏｍｐｕｔａｔｉｏｎｏｆＳｈａｌｌｏｗ－ｗａｔｅｒＭｏｄｅｌｏｎＷｏｒｋｓｔａｔｉｏｎｓＣｌｕｓｔｅｒＳｏｎｇＪｕｎｑｉａｎｇ；ＳｕｎＡｎ－ｃｌａｎｇ，；ＬｉＸｉａｏｍｅｉ（ｅｐａｒｔｍｅｎｔＯｆＣＯ，ｍｐ＇ｕｔｅｒＳｃｉｅｎｃｅ，...

Global SH-wave propagation in a 2D whole Moon model using the parallel hybrid PSM/FDM method

We present numerical modeling of SH-wave propagation for the recently proposed whole Moon model and try to improve our understanding of lunar seismic wave propagation. We use a hybrid PSM/FDM method on staggered grids to solve the wave equations and implement the calculation on a parallel PC cluster to improve the computing efficiency. Features of global SH-wave propagation are firstly discussed for a 100-km shallow and900-km deep moonquakes, respectively. Effects of frequency range and lateral variation of crust thickness are then investigated with various models. Our synthetic waveforms are finally compared with observed Apollo data to show the features of wave propagation that were produced by our model and those not reproduced by our models. Our numerical modeling show that the low-velocity upper crust plays significant role in the development of reverberating wave trains. Increasing frequency enhances the strength and duration of the reverberations.Surface multiples dominate wavefields for shallow event.Core–mantle reflections can be clearly identified for deep event at low frequency. The layered whole Moon model and the low-velocity upper crust produce the reverberating wave trains following each phases consistent with observation. However, more realistic Moon model should be considered in order to explain the strong and slow decay scattering between various phases shown on observation data.

Block and parallel modelling of broad domain nonlinear continuous mapping based on NN

The necessity of the use of the block and parallel modeling of the nonlinear continuous mappings with NN is firstly expounded quantitatively. Then, a practical approach for the block and parallel modeling of the nonlinear continuous mappings with NN is proposed. Finally, an example indicating that the method raised in this paper can be realized by suitable existed software is given. The results of the experiment of the model discussed on the 3-D Mexican straw hat indicate that the block and parallel modeling based on NN is more precise and faster in computation than the direct ones and it is obviously a concrete example and the development of the large-scale general model established by Tu Xuyan.

Designing and optimizing a parallel neural network model for predicting the solubility of diosgenin in n-alkanols

Accurate estimation of the solubility of a chemical compound is an important issue for many industrial proce sses.To overcome the defects of some thermodynamic models and simple correlations,a parallel neural network(PNN) model was conceived and optimized to predict the solubility of diosgenin in seven n-alkanols(C_(1)-C_(7)).The linear regression analysis of the parity plots indicates that the PNN model can give more accurate descriptions of the solubility of diosgenin than the ordinary neural network(ONN) model.The comparison of the average root mean square deviation(RMSD) shows that the suggested model has a slight advantage over the thermodynamic NRTL model in terms of the calculating precision.Moreover,the PNN model can reflect the effects of the temperature and the chain length of the alcohol solvent on the solution behavior of diosgenin correctly and can estimate its solubility in the n-alkanols with more carbon atoms.

Water Transport Models of Moisture Absorption and Sweat Discharge Yarns

An important property of moisture absorption and sweat discharge yarns is their water transport property. In the paper, two water transport models of moisture absorption and sweat discharge yarns were developed to investigate the influence factors on their wicking rate. In parallel Column Pores Model, wicking rate is determined by the equivalent capillary radius R and length of the capillary tube L. In Pellets Accumulation Model, wicking rate is decided by the capillary radius r and length of the fiber unit assemble L0.

DISTRIBUTION OF REACHABLE WORKSPACE AREAS IN PHYSICAL MODEL OF THE SOLUTION SPACE OF 2-DOF PARALLEL PLANAR MANIPULATORS

The expressions for calculating the values of the workspace areas of 2-DOF parallel planar manipulators (PPM) is derived. By the aid of computer, the values are calculated and plotted on the physical model of the solution space of the 2-DOF PPMs,so the workspaee-area-property atlas is obtained. The atlas delineates the relationship between the workspace areas and the link lengths of the 2-DOF PPMs all-sidedly. It is very useful for designers overalI to understand and know welI the relationship.

Vertical human-induced vibration of pedestrian bridge

The double degrees-of-freedom（DOFs）parallel model is adopted to analyze static vertical human-induced vibration with the finite element analysis（FEA）method.In the first-order symmetric vibration mode,the periods of the spring-mass model gradually decrease with the increase in K1 and K2,but they are always greater than the period of the add-on mass model.Meanwhile,the periods of the spring-mass model decrease with the decrease in m1 and m2,but they are always greater than the period of the hollow bridge model.Since the human＇s two degrees-of-freedom vibrate in the same direction as that of the bridge mid-span,the existence of human＇s rigidity leads to the reduction in the rigidity of the spring-mass model.In the second-order symmetric vibration mode,the changes of rigidity K2 and mass m2 result in the disappearance or occurrence of some vibration modes.It can be concluded that compared with the spring-mass model,the results of the add-on mass model lean to lack of safety to the structure;besides,the DOF with a smaller ratio of mass to rigidity plays the chief role in the vibration of the structure.

Construction of tree volume equations for Chinese fir plantations in Guizhou Province, southwestern China

Forest volume, the major component of forest biomass, is an important issue in forest resource monitoring.It is estimated from tree volume tables or equations. Based on tree volume data of 1840 sample trees from Chinese fir （Cunninghamia lanceolata） plantations in Guizhou Province in southwestern China, parallel one- and two-variable tree volume tables and tree height curves for central and other areas were constructed using an error-in-variable modeling method. The results show that, although the one-variable tree volume equations and height curves between the central and other areas were significantly different, the two-variable volume equations were sufficiently close, so that a generalized two-variable tree volume equation could be established for the entire province.

On the Annual Cycle Characteristics of the Sea Surface Height in South China Sea

The annual cycle characteristics of the SSH in the South China Sea (SCS) are analyzed based on the Sea Surface Height (SSH) anomaly data from the TOPEX / POSEIDON-ERS altimeter data and the Parallel Ocean Climate Model (POCM) prediction. The results show that the distributions of the SSH anomalies of the SCS in January, March and May, are opposite to those in July, September and November respectively; In January (July) there is the SSH negative (positive) anomaly in the deep water basin and at the Luzon Strait, while there is positive (negative) anomaly on the most of continental shelves in the west and south of South China Sea; In March (September) the SSH anomalies are similar to those in January (July), although their magnitudes have decreased and a small positive (negative) anomaly appears in the center of the South China Sea; The amplitude of the SSH annual cycle reaches its maximum in the Northwest of the Luzon Island; The seasonal variability of the wind stress is dominant in the formation of the SSH seasonal variability.

A New Study on Combustion Behavior of Pine Sawdust Characterized by the Weibull Distribution

An isothermal operation is implemented by employing a thermogravimetric analyzer (TGA) for simulating the thermal decomposition behavior of 58μm pine sawdust in air atmosphere.An independent parallel reaction model is adopted in this study to describe the thermal decomposition mechanism.The Weibull distribution function is used to record and analyze the weight loss during isothermal decomposition at different temperatures(500,600,700,and 800°C).The total weight loss of the pine sawdust is assumed as a linear combination of individual weight loss from three components,including the char and two volatile matters.The plot of the thermal decomposition rate curve leads to kinetic parameters such as the reaction rate constants and the reaction order.The results show that the Weibull distribution function successfully represents decomposition curves of three components,and fits the experimental data very well.Therefore,this study provides a simple way to evaluate the decomposition rate of biomass combustion in a real combustor.

Quantum Homomorphic Signature with Repeatable Verification

In January 2015,the first quantum homomorphic signature scheme was proposed creatively.However,only one verifier is allowed to verify a signature once in this scheme.In order to support repeatable verification for general scenario,we propose a new quantum homomorphic signature scheme with repeatable verification by introducing serial verification model and parallel verification model.Serial verification model solves the problem of signature verification by combining key distribution and Bell measurement.Parallel verification model solves the problem of signature duplication by logically treating one particle of an EPR pair as a quantum signature and physically preparing a new EPR pair.These models will be beneficial to the signature verification of general scenarios.Scheme analysis shows that both intermediate verifiers and terminal verifiers can successfully verify signatures in the same operation with fewer resource consumption,and especially the verified signature in entangled states can be used repeatedly.

Fast Parallel Algorithm for Slicing STL Based on Pipeline

In Additive Manufacturing field, the current researches of data processing mainly focus on a slicing process of large STL files or complicated CAD models. To improve the efficiency and reduce the slicing time, a parallel algorithm has great advantages. However, traditional algorithms can＇t make full use of multi-core CPU hardware resources. In the paper, a fast parallel algorithm is presented to speed up data processing. A pipeline mode is adopted to design the parallel algorithm. And the complexity of the pipeline algorithm is analyzed theoretically. To evaluate the performance of the new algorithm, effects of threads number and layers number are investigated by a serial of experiments. The experimental results show that the threads number and layers number are two remarkable factors to the speedup ratio. The tendency of speedup versus threads number reveals a positive relationship which greatly agrees with the Amdahl＇s law, and the tendency of speedup versus layers number also keeps a positive relationship agreeing with Gustafson＇s law. The new algorithm uses topological information to compute contours with a parallel method of speedup. Another parallel algorithm based on data parallel is used in experiments to show that pipeline parallel mode is more efficient. A case study at last shows a suspending performance of the new parallel algorithm. Compared with the serial slicing algorithm, the new pipeline parallel algorithm can make full use of the multi-core CPU hardware, accelerate the slicing process, and compared with the data parallel slicing algorithm, the new slicing algorithm in this paper adopts a pipeline parallel model, and a much higher speedup ratio and efficiency is achieved.

Neurodynamics analysis of brain information transmission

This paper proposes a model of neural networks consisting of populations of perceptive neurons, inter-neurons, and motor neurons according to the theory of stochastic phase resetting dynamics. According to this model, the dynamical characteristics of neural networks are studied in three coupling cases, namely, series and parallel coupling, series coupling, and unilateral coupling. The results show that the indentified structure of neural networks enables the basic characteristics of neural information processing to be described in terms of the actions of both the optional motor and the reflected motor. The excitation of local neural networks is caused by the action of the optional motor. In particular, the excitation of the neural population caused by the action of the optional motor in the motor cortex is larger than that caused by the action of the reflected motor. This phenomenon indicates that there are more neurons participating in the neural information processing and the excited synchronization motion under the action of the optional motor.