Cloud-Edge Collaborative Federated GAN Based Data Processing for IoT-Empowered Multi-Flow Integrated Energy Aggregation Dispatch

The convergence of Internet of Things(IoT),5G,and cloud collaboration offers tailored solutions to the rigorous demands of multi-flow integrated energy aggregation dispatch data processing.While generative adversarial networks(GANs)are instrumental in resource scheduling,their application in this domain is impeded by challenges such as convergence speed,inferior optimality searching capability,and the inability to learn from failed decision making feedbacks.Therefore,a cloud-edge collaborative federated GAN-based communication and computing resource scheduling algorithm with long-term constraint violation sensitiveness is proposed to address these challenges.The proposed algorithm facilitates real-time,energy-efficient data processing by optimizing transmission power control,data migration,and computing resource allocation.It employs federated learning for global parameter aggregation to enhance GAN parameter updating and dynamically adjusts GAN learning rates and global aggregation weights based on energy consumption constraint violations.Simulation results indicate that the proposed algorithm effectively reduces data processing latency,energy consumption,and convergence time.

Optimizing Healthcare Big Data Processing with Containerized PySpark and Parallel Computing: A Study on ETL Pipeline Efficiency

In this study, we delve into the realm of efficient Big Data Engineering and Extract, Transform, Load (ETL) processes within the healthcare sector, leveraging the robust foundation provided by the MIMIC-III Clinical Database. Our investigation entails a comprehensive exploration of various methodologies aimed at enhancing the efficiency of ETL processes, with a primary emphasis on optimizing time and resource utilization. Through meticulous experimentation utilizing a representative dataset, we shed light on the advantages associated with the incorporation of PySpark and Docker containerized applications. Our research illuminates significant advancements in time efficiency, process streamlining, and resource optimization attained through the utilization of PySpark for distributed computing within Big Data Engineering workflows. Additionally, we underscore the strategic integration of Docker containers, delineating their pivotal role in augmenting scalability and reproducibility within the ETL pipeline. This paper encapsulates the pivotal insights gleaned from our experimental journey, accentuating the practical implications and benefits entailed in the adoption of PySpark and Docker. By streamlining Big Data Engineering and ETL processes in the context of clinical big data, our study contributes to the ongoing discourse on optimizing data processing efficiency in healthcare applications. The source code is available on request.

Status of UnDifferenced and Uncombined GNSS Data Processing Activities in China

With the continued development of multiple Global Navigation Satellite Systems(GNSS)and the emergence of various frequencies,UnDifferenced and UnCombined(UDUC)data processing has become an increasingly attractive option.In this contribution,we provide an overview of the current status of UDUC GNSS data processing activities in China.These activities encompass the formulation of Precise Point Positioning(PPP)models and PPP-Real-Time Kinematic(PPP-RTK)models for processing single-station and multi-station GNSS data,respectively.Regarding single-station data processing,we discuss the advancements in PPP models,particularly the extension from a single system to multiple systems,and from dual frequencies to single and multiple frequencies.Additionally,we introduce the modified PPP model,which accounts for the time variation of receiver code biases,a departure from the conventional PPP model that typically assumes these biases to be time-constant.In the realm of multi-station PPP-RTK data processing,we introduce the ionosphere-weighted PPP-RTK model,which enhances the model strength by considering the spatial correlation of ionospheric delays.We also review the phase-only PPP-RTK model,designed to mitigate the impact of unmodelled code-related errors.Furthermore,we explore GLONASS PPP-RTK,achieved through the application of the integer-estimable model.For large-scale network data processing,we introduce the all-in-view PPP-RTK model,which alleviates the strict common-view requirement at all receivers.Moreover,we present the decentralized PPP-RTK data processing strategy,designed to improve computational efficiency.Overall,this work highlights the various advancements in UDUC GNSS data processing,providing insights into the state-of-the-art techniques employed in China to achieve precise GNSS applications.

A novel technique for automatic seismic data processing using both integral and local feature of seismograms

A novel technique for automatic seismic data processing using both integral and local feature of seismograms was presented in this paper. Here, the term integral feature of seismograms refers to feature which may depict the shape of the whole seismograms. However, unlike some previous efforts which completely abandon the DIAL approach, i.e., signal detection, phase identifi- cation, association, and event localization, and seek to use envelope cross-correlation to detect seismic events directly, our technique keeps following the DIAL approach, but in addition to detect signals corresponding to individual seismic phases, it also detects continuous wave-trains and explores their feature for phase-type identification and signal association. More concrete ideas about how to define wave-trains and combine them with various detections, as well as how to measure and utilize their feature in the seismic data processing were expatiated in the paper. This approach has been applied to the routine data processing by us for years, and test results for a 16 days＇ period using data from the Xinjiang seismic station network were presented. The automatic processing results have fairly low false and missed event rate simultaneously, showing that the new technique has good application prospects for improvement of the automatic seismic data processing.

Cost of Multicast Logical Key Tree Based on Hierarchical Data Processing

How to design a multicast key management system with high performance is a hot issue now. This paper will apply the idea of hierarchical data processing to construct a common analytic model based on directed logical key tree and supply two important metrics to this problem： re-keying cost and key storage cost. The paper gives the basic theory to the hierarchical data processing and the analyzing model to multieast key management based on logical key tree. It has been proved that the 4-ray tree has the best performance in using these metrics. The key management problem is also investigated based on user probability model, and gives two evaluating parameters to re-keying and key storage cost.

GF-3 data real-time processing method based on multi-satellite distributed data processing system

Due to the limited scenes that synthetic aperture radar(SAR)satellites can detect,the full-track utilization rate is not high.Because of the computing and storage limitation of one satellite,it is difficult to process large amounts of data of spaceborne synthetic aperture radars.It is proposed to use a new method of networked satellite data processing for improving the efficiency of data processing.A multi-satellite distributed SAR real-time processing method based on Chirp Scaling(CS)imaging algorithm is studied in this paper,and a distributed data processing system is built with field programmable gate array(FPGA)chips as the kernel.Different from the traditional CS algorithm processing,the system divides data processing into three stages.The computing tasks are reasonably allocated to different data processing units(i.e.,satellites)in each stage.The method effectively saves computing and storage resources of satellites,improves the utilization rate of a single satellite,and shortens the data processing time.Gaofen-3(GF-3)satellite SAR raw data is processed by the system,with the performance of the method verified.

SLC-index: A scalable skip list-based index for cloud data processing

Due to the increasing number of cloud applications,the amount of data in the cloud shows signs of growing faster than ever before.The nature of cloud computing requires cloud data processing systems that can handle huge volumes of data and have high performance.However,most cloud storage systems currently adopt a hash-like approach to retrieving data that only supports simple keyword-based enquiries,but lacks various forms of information search.Therefore,a scalable and efficient indexing scheme is clearly required.In this paper,we present a skip list-based cloud index,called SLC-index,which is a novel,scalable skip list-based indexing for cloud data processing.The SLC-index offers a two-layered architecture for extending indexing scope and facilitating better throughput.Dynamic load-balancing for the SLC-index is achieved by online migration of index nodes between servers.Furthermore,it is a flexible system due to its dynamic addition and removal of servers.The SLC-index is efficient for both point and range queries.Experimental results show the efficiency of the SLC-index and its usefulness as an alternative approach for cloud-suitable data structures.

Effects of the differences between the ITRF2000 and ITRF2005 models in GNSS data processing

In comparison with the ITRF2000 model, the ITRF2005 model represents a significant improvement in solution generation, datum definition and realization. However, these improvements cause a frame difference between the ITRF2000 and ITRF2005 models, which may impact GNSS data processing. To quantify this im- pact, the differences of the GNSS results obtained using the two models, including station coordinates, base- line length and horizontal velocity field, were analyzed. After transformation, the differences in position were at the millimeter level, and the differences in baseline length were less than 1 ram. The differences in the hori- zontal velocity fields decreased with as the study area was reduced. For a large region, the differences in these value were less than 1 mm/a, with a systematic difference of approximately 2 degrees in direction, while for a medium-sized region, the differences in value and direction were not significant.

APPLICATION OF NOISE REDUCTION METHOD BASED ON CURVELET THRESHOLDING NEURAL NETWORK FOR POLAR ICE RADAR DATA PROCESSING

Due to the demand of data processing for polar ice radar in our laboratory, a Curvelet Thresholding Neural Network (TNN) noise reduction method is proposed, and a new threshold function with infinite-order continuous derivative is constructed. The method is based on TNN model. In the learning process of TNN, the gradient descent method is adopted to solve the adaptive optimal thresholds of different scales and directions in Curvelet domain, and to achieve an optimal mean square error performance. In this paper, the specific implementation steps are presented, and the superiority of this method is verified by simulation. Finally, the proposed method is used to process the ice radar data obtained during the 28th Chinese National Antarctic Research Expedition in the region of Zhongshan Station, Antarctica. Experimental results show that the proposed method can reduce the noise effectively, while preserving the edge of the ice layers.

Seismic Data Processing Approaches for the Study of Gas Hydrates in the East China Sea

A comprehensive study of the data profiles, including the 2D seismic data, single channel seismic data, shallow sections, etc., reveals that gas hydrates occur in the East China Sea. A series of special techniques are used in the processing of seismic data, which include enhancing the accuracy of velocity analysis and resolution, estimating the wavelet, suppressing the multiple, preserving the relative amplitude, using the DMO and AVO techniques and some special techniques in dealing with the wave impedance. The existence of gas hydrates is reflected in the subbottom profiles in the appearance of BSRs, amplitude anomalies, velocity anomalies and AVO anomalies, etc. Hence the gas hydrates can be identified and predicted. It is pointed out that the East China Sea is a favorable area of the gas hydrates resource, and the Okinawa Trough is a target area of gas hydrates reservoir.

Seismic Data Processing and Interpretation on the Loess Plateau, Part1: Seismic Data Processing

Branching river channels and the coexistence of valleys, ridges, hiils, and slopes＇as the result of long-term weathering and erosion form the unique loess topography. The Changqing Geophysical Company, working in these complex conditions, has established a suite of technologies for high-fidelity processing and fine interpretation of seismic data. This article introduces the processes involved in the data processing and interpretation and illustrates the results.

Reconfigurable-System-on-Chip Implementation of Data Processing Units for Space Applications

Application-specific data processing units (DPUs) are commonly adopted for operational control and data processing in space missions. To overcome the limitations of traditional radiation-hardened or fully commercial design approaches, a reconfigurable-system-on-chip (RSoC) solution based on state-of-the-art FPGA is introduced. The flexibility and reliability of this approach are outlined, and the requirements for an enhanced RSoC design with in-flight reconfigurability for space applications are presented. This design has been demonstrated as an on-board computer prototype, providing an in-flight reconfigurable DPU design approach using integrated hardwired processors.

Application and evaluation of layering shear method in LADCP data processing

The current velocity observation of LADCP(Lowered Acoustic Doppler Current Profiler)has the advantages of a large vertical range of observation and high operability compared with traditional current measurement methods,and is being widely used in the field of ocean observation.Shear and inverse methods are now commonly used by the international marine community to process LADCP data and calculate ocean current profiles.The two methods have their advantages and shortcomings.The shear method calculates the value of current shear more accurately,while the accuracy in an absolute value of the current is lower.The inverse method calculates the absolute value of the current velocity more accurately,but the current shear is less accurate.Based on the shear method,this paper proposes a layering shear method to calculate the current velocity profile by“layering averaging”,and proposes corresponding current calculation methods according to the different types of problems in several field observation data from the western Pacific,forming an independent LADCP data processing system.The comparison results have shown that the layering shear method can achieve the same effect as the inverse method in the calculation of the absolute value of current velocity,while retaining the advantages of the shear method in the calculation of a value of the current shear.

Application of Big Data Technology in Evaluation of Operating Status of High-pressure Hydrogenation Heat Exchanger

The high-pressure hydrogenation heat exchanger is an impoltmlt equipment of the refinery, but it is exposed tothe problem of leakage caused by ammonium salt corrosion. Therefore, it is very important to evaluate the operating statusof flae hydrogenation heat exchanger. To improve flae method for evaluating the operating status of hydrogenation heat ex-chmagers by using flae traditional method, flais paper proposes a new method for evaluating the operation of hydrogenationheat exchangers based on big data. To address flae noisy data common in flae industry, this paper proposes an automatednoisy interval detection algorithm. To deal with flae problem that the sensor parameters have voluminous and mtrelateddimensions, flais paper proposes a key parameter detection algorithm based on flae Pearson correlation coefficient. Finally,this paper presents a system-based health scoring algorithm based on PCA （Principal Component Analysis） to assist site op-erators in assessing the healfla of hydrogenation heat exchangers. The evaluation of flae operating status of flae hydrorefiningheat exchange device based on big data technology will help the operators to more accurately grasp the status of flae indus-trial system mad have positive guiding significance for the early warning offlae failure.

Integrated Data Processing Method for GPS and INS Field Test over Rocky Mountain

The method of integrated data processing for GPS and INS(inertial navigation system) field test over the Rocky Mountains using the adaptive Kalman filtering technique is presented. On the basis of the known GPS outputs and the offset of GPS and INS, state equations and observations are designed to perform the calculation and improve the navigation accuracy. An example shows that with the method the reliable navigation parameters have been obtained.

Fast Data Processing of a Polarimeter-Interferometer System on J-TEXT

A method of fast data processing has been developed to rapidly obtain evolution of the electron density profile for a multichannel polarimeter-interferometer system（POLARIS）on J-TEXT. Compared with the Abel inversion method, evolution of the density profile analyzed by this method can quickly offer important information. This method has the advantage of fast calculation speed with the order of ten milliseconds per normal shot and it is capable of processing up to 1 MHz sampled data, which is helpful for studying density sawtooth instability and the disruption between shots. In the duration of a flat-top plasma current of usual ohmic discharges on J-TEXT, shape factor u is ranged from 4 to 5. When the disruption of discharge happens, the density profile becomes peaked and the shape factor u typically decreases to 1.

Development and application of geochemical data processing system based on fractal theory

GC-GIS system is a geochemical data processing system based on fractal theory. The system realized quantity statistics function by calling Surfer and MapInfo software, and it is compiled with Visual Basic language. This system is designed to integrate the functions both quantity statistics of Surfer and spatial data management of MapInfo. A new algorithm of fractal is added up to GC-GIS. Taking example for Weichang region of Hebei to test the system, the processing results show that the model can match the real distribution of mine well.

Combining reinforcement learning with mathematical programming:An approach for optimal design of heat exchanger networks

Heat integration is important for energy-saving in the process industry.It is linked to the persistently challenging task of optimal design of heat exchanger networks(HEN).Due to the inherent highly nonconvex nonlinear and combinatorial nature of the HEN problem,it is not easy to find solutions of high quality for large-scale problems.The reinforcement learning(RL)method,which learns strategies through ongoing exploration and exploitation,reveals advantages in such area.However,due to the complexity of the HEN design problem,the RL method for HEN should be dedicated and designed.A hybrid strategy combining RL with mathematical programming is proposed to take better advantage of both methods.An insightful state representation of the HEN structure as well as a customized reward function is introduced.A Q-learning algorithm is applied to update the HEN structure using theε-greedy strategy.Better results are obtained from three literature cases of different scales.

Research on the Application of Intelligence Teaching Mode of Superstar Platform in the Teaching of Experimental Design and Data Processing

Experimental Design and Data Processing is an important core professional basic course for food science majors.This course is theoretical and practical,and there are many formulas,abstract contents and difficult to understand,and there are some problems in the teaching process,such as students1 poor interest in learning,insufficient mastery of what they have learned,and inability to combine theory with practice organically.Through analyzing the existing problems,this paper puts forward some reform measures for the teaching mode of experimental design and data processing by using the intelligent teaching of Superstar platform.

Magnetic field data processing methods of the China Seismo-Electromagnetic Satellite

The High Precision Magnetometer(HPM) on board the China Seismo-Electromagnetic Satellite(CSES) allows highly accurate measurement of the geomagnetic field; it includes FGM(Fluxgate Magnetometer) and CDSM(Coupled Dark State Magnetometer)probes. This article introduces the main processing method, algorithm, and processing procedure of the HPM data. First, the FGM and CDSM probes are calibrated according to ground sensor data. Then the FGM linear parameters can be corrected in orbit, by applying the absolute vector magnetic field correction algorithm from CDSM data. At the same time, the magnetic interference of the satellite is eliminated according to ground-satellite magnetic test results. Finally, according to the characteristics of the magnetic field direction in the low latitude region, the transformation matrix between FGM probe and star sensor is calibrated in orbit to determine the correct direction of the magnetic field. Comparing the magnetic field data of CSES and SWARM satellites in five continuous geomagnetic quiet days, the difference in measurements of the vector magnetic field is about 10 nT, which is within the uncertainty interval of geomagnetic disturbance.