A Novel Approach to Design Distribution Preserving Framework for Big Data

In several fields like financial dealing,industry,business,medicine,et cetera,Big Data(BD)has been utilized extensively,which is nothing but a collection of a huge amount of data.However,it is highly complicated along with time-consuming to process a massive amount of data.Thus,to design the Distribution Preserving Framework for BD,a novel methodology has been proposed utilizing Manhattan Distance(MD)-centered Partition Around Medoid(MD–PAM)along with Conjugate Gradient Artificial Neural Network(CG-ANN),which undergoes various steps to reduce the complications of BD.Firstly,the data are processed in the pre-processing phase by mitigating the data repetition utilizing the map-reduce function;subsequently,the missing data are handled by substituting or by ignoring the missed values.After that,the data are transmuted into a normalized form.Next,to enhance the classification performance,the data’s dimensionalities are minimized by employing Gaussian Kernel(GK)-Fisher Discriminant Analysis(GK-FDA).Afterwards,the processed data is submitted to the partitioning phase after transmuting it into a structured format.In the partition phase,by utilizing the MD-PAM,the data are partitioned along with grouped into a cluster.Lastly,by employing CG-ANN,the data are classified in the classification phase so that the needed data can be effortlessly retrieved by the user.To analogize the outcomes of the CG-ANN with the prevailing methodologies,the NSL-KDD openly accessible datasets are utilized.The experiential outcomes displayed that an efficient result along with a reduced computation cost was shown by the proposed CG-ANN.The proposed work outperforms well in terms of accuracy,sensitivity and specificity than the existing systems.

Distributed adaptive direct position determination based on diffusion framework

The conventional direct position determination（DPD） algorithm processes all received signals on a single sensor.When sensors have limited computational capabilities or energy storage,it is desirable to distribute the computation among other sensors.A distributed adaptive DPD（DADPD）algorithm based on diffusion framework is proposed for emitter localization.Unlike the corresponding centralized adaptive DPD（CADPD） algorithm,all but one sensor in the proposed algorithm participate in processing the received signals and estimating the common emitter position,respectively.The computational load and energy consumption on a single sensor in the CADPD algorithm is distributed among other computing sensors in a balanced manner.Exactly the same iterative localization algorithm is carried out in each computing sensor,respectively,and the algorithm in each computing sensor exhibits quite similar convergence behavior.The difference of the localization and tracking performance between the proposed distributed algorithm and the corresponding CADPD algorithm is negligible through simulation evaluations.

Framework of Distributed Coupled Atmosphere-Ocean-Wave Modeling System

为了在沿海的区域的集中的天气系统研究在空气和海洋以及他们的重要角色之间的相互作用，并且改进危险天气的预报能力，沿海的区域处理，为系统建模的 coupledatmosphere-ocean-wave 被开发了。为连接模型的基于代理人的环境框架允许在模型之间的灵活、动态的信息交换。为灵活性，可移植性和可伸缩性的目的，整个系统的框架拿包括用户接口层，计算的层和服务创新的层的 multi-layerarchitecture。在这篇论文介绍的数字实验表明分布式的联合当模特儿的系统的性能。

A Tutorial on Federated Learning from Theory to Practice:Foundations,Software Frameworks,Exemplary Use Cases,and Selected Trends

When data privacy is imposed as a necessity,Federated learning(FL)emerges as a relevant artificial intelligence field for developing machine learning(ML)models in a distributed and decentralized environment.FL allows ML models to be trained on local devices without any need for centralized data transfer,thereby reducing both the exposure of sensitive data and the possibility of data interception by malicious third parties.This paradigm has gained momentum in the last few years,spurred by the plethora of real-world applications that have leveraged its ability to improve the efficiency of distributed learning and to accommodate numerous participants with their data sources.By virtue of FL,models can be learned from all such distributed data sources while preserving data privacy.The aim of this paper is to provide a practical tutorial on FL,including a short methodology and a systematic analysis of existing software frameworks.Furthermore,our tutorial provides exemplary cases of study from three complementary perspectives:i)Foundations of FL,describing the main components of FL,from key elements to FL categories;ii)Implementation guidelines and exemplary cases of study,by systematically examining the functionalities provided by existing software frameworks for FL deployment,devising a methodology to design a FL scenario,and providing exemplary cases of study with source code for different ML approaches;and iii)Trends,shortly reviewing a non-exhaustive list of research directions that are under active investigation in the current FL landscape.The ultimate purpose of this work is to establish itself as a referential work for researchers,developers,and data scientists willing to explore the capabilities of FL in practical applications.

A Fully Distributed Hybrid Control Framework For Non-Differentiable Multi-Agent Optimization

This paper develops a fully distributed hybrid control framework for distributed constrained optimization problems.The individual cost functions are non-differentiable and convex.Based on hybrid dynamical systems,we present a distributed state-dependent hybrid design to improve the transient performance of distributed primal-dual first-order optimization methods.The proposed framework consists of a distributed constrained continuous-time mapping in the form of a differential inclusion and a distributed discrete-time mapping triggered by the satisfaction of local jump set.With the semistability theory of hybrid dynamical systems,the paper proves that the hybrid control algorithm converges to one optimal solution instead of oscillating among different solutions.Numerical simulations illustrate better transient performance of the proposed hybrid algorithm compared with the results of the existing continuous-time algorithms.

Formalized Description of Distributed Denial of Service Attack

The distributed denial of service (DDoS) attack is one of the dangers in intrusion modes. It's difficult to defense and can cause serious damage to the system. Based on a careful study of the attack principles and characteristics, an object-oriented formalized description is presented, which contains a three-level framework and offers full specifications of all kinds of DDoS modes and their features and the relations between one another. Its greatest merit lies in that it contributes to analyzing, checking and judging DDoS. Now this formalized description has been used in a special IDS and it works very effectively.(

Power loss reduction of distribution systems using BFO based optimal reconfiguration along with DG and shunt capacitor placement simultaneously in fuzzy framework

In distribution systems,network reconfiguration and capacitor placement are commonly used to diminish power losses and keep voltage profiles within acceptable limits.Moreover,the problem of DG allocation and sizing is great important.In this work,a combination of a fuzzy multi-objective approach and bacterial foraging optimization(BFO) as a meta-heuristic algorithm is used to solve the simultaneous reconfiguration and optimal sizing of DGs and shunt capacitors in a distribution system.Each objective is transferred into fuzzy domain using its membership function.Then,the overall fuzzy satisfaction function is formed and considered a fitness function inasmuch as the value of this function has to be maximized to gain the optimal solution.The numerical results show that the presented algorithm improves the performance much more than other meta-heuristic algorithms.Simulation results found that simultaneous reconfiguration with DG and shunt capacitors allocation(case 5) has 77.41%,42.15%,and 56.14%improvements in power loss reduction,load balancing,and voltage profile indices,respectively in 33-bus test system.This result found 87.27%,35.82%,and 54.34%improvements of mentioned indices respectively for 69-bus system.

县域休闲农业体验产品规划技术及实证——以全国休闲农业重点县临安区为例

休闲农业是推进乡村产业振兴和农文旅融合的重要产业和关键业态。县域是政策制度设计与空间落地实施的重要行政单元,更是消费者休闲农业体验的重要目的地。因此,从县域尺度开展休闲农业体验产品规划是促进休闲农业转型升级的关键路径。本研究引入昂谱(RMP)理论和体验经济理论,基于改进的E-RMP分析框架,构建了县域休闲农业体验产品规划技术流程,并以全国休闲农业重点县浙江省杭州市临安区为例,在资源转化评价、市场体验评价、产品创新外部环境分析的基础上,对全区休闲农业体验产品空间布局、产品谱系和游线组织等方面提出规划策略。空间布局上,全区构建了一条杭黄世界级山地特色农业景观廊道和东部都市休闲农业、中部耕织体验农业、西部山地特色农业三个组团“1+3”格局,并规划出不同主题、不同季节的休闲农业体验产品谱系和游览线路。E-RMP分析框架为县域尺度开展休闲农业体验产品规划提供技术遵循,为合理开展空间布局、产品谱系和游览线路设计提供技术支撑。本研究为未来几年地方开展县域休闲农业规划和争创全国休闲农业重点县提供经验借鉴和决策依据。

长江经济带农业经济韧性的时空分布及组态影响因素——基于TOE框架的fs-QCA分析

新发展格局下,增强农业经济韧性是助力农业强国建设的时代需求。从3个维度构建农业经济韧性评价体系,运用熵值法测算2012—2021长江经济带农业经济韧性。借助H-P滤波分析法、标准差椭圆模型、局域热点分析描绘农业经济韧性的时空演化格局。基于组态视角,采用fsQCA并依托TOE理论框架,厘清长江经济带农业经济韧性的多元提升路径。结果发现:长江经济带农业经济韧性共经历2轮周期波动,周期波动态势有所强化。长江经济带农业经济韧性重心向东南方向移动,发展日益均衡化。热点区和次热点区空间集聚范围主要分布于长三角地区。组态视角下,存在4种组态路径实现农业经济韧性的提升,分别是融智拉动型、要素提质型、技术主导产业结构支持型和全面优化型。

考虑天气特征与多变量相关性的配电网短期负荷预测

针对配电网短期负荷预测受到众多复杂天气特征等随机不确定性因素影响,以及传统预测模型难以有效分析不同特征序列之间的相关性等问题,提出一种考虑天气特征与多变量相关性的配电网短期负荷预测方法。首先,提出多变量快速最大信息系数(multi-variable rapid maximal information coefficient,MVRapidMIC)提取相关性高的天气特征序列。其次,引入探索性因子分析法(exploratory factor analysis,EFA),对高相关性特征序列进行降维处理。最后,将维度分段(dimension-segment-wise,DSW)机制和两阶段注意力(two-stage attention,TSA)机制与Informer模型结合,提高预测模型对不同特征序列相关性的分析能力。通过DTU 7K 47节点实际配电网的历史负荷数据开展仿真测试,验证所提方法的预测精度、鲁棒性和时效性。

考虑群体决策差异冲突解决机制的配电站房健康状态评估方法

针对配电站房缺乏健康评估机制、运维周期设置不合理的问题,提出了一种考虑群体决策差异冲突解决机制的配电站房健康状态综合评估方法。首先,建立配电站房指标体系和专家评价框架,设计了一种新型的二元冲突测量函数来量化全局冲突。然后,使用专家评价结果的虚假度、可信度、可用度等测度指标构造专家修正因子,以改进D-S证据理论,通过聚合不同专家的评价意见来量化评价指标的权重。接着,建立改进灰色关联度-逼近理想解法(grey relation analysis-technique for order preference by similarity to an ideal solution, GRA-TOPSIS)评估模型,引入灰色关联接近度,与距离接近度融合得到综合接近度,改善TOPSIS评价判据片面性的缺陷。最后,计算每个配电站房的评价值与理想解之间的综合接近度,反映配电站房的健康状态。实验分析表明该方法能兼容专家评价之间的冲突性、差异性、不确定性,与现有方法相比评估结果更具准确性和合理性,对运维人员制定合理的检修决策具有一定的指导价值。

毛白杨木材/MOF-5复合材料的制备及甲苯吸附性能研究

优化MOF-5的合成工艺,确定金属中心与有机配体的最佳配比,并基于最佳配比通过真空浸渍和水热作用在毛白杨木材孔隙内部原位合成MOF-5,制备毛白杨木材/MOF-5复合材料,并对毛白杨木材/MOF-5复合材料的微观形貌、结合机理、孔隙结构和甲苯吸附性能进行表征。结果表明,当金属中心与有机配体物质的量比为1∶3时,合成的MOF-5晶体粒径较小,BET比表面积为268.729 m^(2)/g,孔容为0.136 cm^(3)/g,且具备典型的MOF-5晶体X射线特征峰。在毛白杨木材孔隙内部原位合成的MOF-5的平均负载量为22.6%;红外光谱的分析结果显示MOF-5与毛白杨木材通过氢键和静电相互作用相结合。扫描电镜、压汞法和氮气吸/脱附测试的孔隙结构表明,MOF-5填充了毛白杨木材中的部分大孔和介孔,增加了微孔的比表面积和孔容。常温常压下毛白杨木材/MOF-5复合材料对甲苯的最大吸附量为16.07 cm^(3)/g,展现了较好的甲苯吸/脱附性能。毛白杨木材/MOF-5复合材料在气体吸附与分离领域展现了较好的应用潜力,这为速生木材作为吸附材料的功能化应用提供了参考。

数据驱动的湿地微塑料分布格局与影响因素研究

湿地作为水生态系重要拦污屏障和关键缓冲带,其不断累积的微塑料污染负荷不仅破坏了其自身生态系统结构和功能,也因独特的生态位“塑料际”增加了水源性传播疾病的扩散。然而,目前尚不清楚湿地微塑料在全球范围内的分布格局及其驱动机制。鉴于此,本研究基于数据汇编、整理和统计分析框架,整合了来自19个湿地的200个微塑料样本数据,以阐释湿地微塑料赋存特征并识别其影响因素。多元统计分析结果表明,湿地类型和环境介质影响微塑料丰度差异,并且内陆湿地的微塑料丰度显著大于海岸湿地。湿地水体和沉积物中微塑料形状和颜色存在显著相似性,并且纤维状的透明微塑料是其主要赋存形态。湿地水体和沉积物中微塑料赋存格局受到微塑料分析方法,包括采样和提取方法的显著影响。此外,地理距离作为主导因素,影响了湿地微塑料的分布,并且观测到微塑料丰度随距离的增加而减少的趋势。本研究基于湿地微塑料数据特征构建的数据驱动框架,既可为更好地掌握湿地微塑料分布格局及其污染状况提供方法参考,也可为湿地甚至其他水生态系统中微塑料的污染管控提供数据支撑。

基于大数据随机样本划分的分布式观测点分类器

观测点分类器(OPC)是一种试图通过将多维样本空间线性不可分问题转换成一维距离空间线性可分问题的有监督学习模型,对高维数据的分类问题尤为有效。针对OPC在处理大数据分类问题时表现的较高训练复杂度,在Spark框架下设计一款基于大数据的随机样本划分(RSP)的分布式OPC(DOPC)。首先,在分布式计算环境下生成大数据的RSP数据块,并将它转换为弹性分布式数据集(RDD);其次,在RSP数据块上协同式地训练一组OPC,由于每个RSP数据块上的OPC独立训练,因此有高效的Spark可实现性;最后,在Spark框架下将在RSP数据块上协同训练的OPC集成为DOPC,对新样本进行类标签预测。在8个大数据集上,对Spark集群环境下实现的DOPC的可行性、合理性和有效性进行实验验证,实验结果显示,DOPC能够以更低的计算消耗获得比单机OPC更高的测试精度,同时相较于Spark框架下实现的基于RSP模型的神经网络(NN)、决策树(DT)、朴素贝叶斯(NB)和K最近邻(KNN),DOPC分类器具有更强的泛化性能。测试结果表明,DOPC是一种高效低耗的处理大数据分类问题的有监督学习算法。

Optimal operation of Internet Data Center with PV and energy storage type of UPS clusters

With the development of green data centers,a large number of Uninterruptible Power Supply(UPS)resources in Internet Data Center(IDC)are becoming idle assets owing to their low utilization rate.The revitalization of these idle UPS resources is an urgent problem that must be addressed.Based on the energy storage type of the UPS(EUPS)and using renewable sources,a solution for IDCs is proposed in this study.Subsequently,an EUPS cluster classification method based on the concept of shared mechanism niche(CSMN)was proposed to effectively solve the EUPS control problem.Accordingly,the classified EUPS aggregation unit was used to determine the optimal operation of the IDC.An IDC cost minimization optimization model was established,and the Quantum Particle Swarm Optimization(QPSO)algorithm was adopted.Finally,the economy and effectiveness of the three-tier optimization framework and model were verified through three case studies.

Cloudless-Training:基于serverless的高效跨地域分布式ML训练框架

跨地域分布式机器学习(ML)训练能够联合多区域的云资源协作训练,可满足许多新兴ML场景(比如大型模型训练、联邦学习)的训练需求。但其训练效率仍受2方面挑战的制约。首先,多区域云资源缺乏有效的弹性调度,这会影响训练的资源利用率和性能;其次,模型跨地域同步需要在广域网(WAN)上高频通信,受WAN的低带宽和高波动的影响,会产生巨大通信开销。本文提出Cloudless-Training,从3个方面实现高效的跨地域分布式ML训练。首先,它基于serverless计算模式实现,使用控制层和训练执行层的2层架构,支持多云区域的弹性调度和通信。其次,它提供一种弹性调度策略,根据可用云资源的异构性和训练数据集的分布自适应地部署训练工作流。最后,它提供了2种高效的跨云同步策略,包括基于梯度累积的异步随机梯度下降(ASGD-GA)和跨云参数服务器(PS)间的模型平均(MA)。Cloudless-Training是基于OpenFaaS实现的,并被部署在腾讯云上评估,实验结果表明Cloudless-Training可显著地提高跨地域分布式ML训练的资源利用率(训练成本降低了9.2%~24.0%)和同步效率(训练速度最多比基线快1.7倍),并能保证模型的收敛精度。

Automatic Generation Control in a Distributed Power Grid Based on Multi-step Reinforcement Learning

The increasing use of renewable energy in the power system results in strong stochastic disturbances and degrades the control performance of the distributed power grids.In this paper,a novel multi-agent collaborative reinforcement learning algorithm is proposed with automatic optimization,namely,Dyna-DQL,to quickly achieve an optimal coordination solution for the multi-area distributed power grids.The proposed Dyna framework is combined with double Q-learning to collect and store the environmental samples.This can iteratively update the agents through buffer replay and real-time data.Thus the environmental data can be fully used to enhance the learning speed of the agents.This mitigates the negative impact of heavy stochastic disturbances caused by the integration of renewable energy on the control performance.Simulations are conducted on two different models to validate the effectiveness of the proposed algorithm.The results demonstrate that the proposed Dyna-DQL algorithm exhibits superior stability and robustness compared to other reinforcement learning algorithms.

基于VSOME/IP的汽车E/E架构分布式服务框架设计研究

新型汽车电子电气架构下的车载软件需具备可复用、易扩展、松耦合、兼容互操作等特点。为了将汽车电子控制单元(ECU)上的应用程序抽象为服务,以开源的分布式通信中间件VSOME/IP和远程过程调用框架CommonAPI C++为基础,提出了一种基于VSOME/IP的分布式服务框架。该框架利用Franca IDL服务接口描述语言提高开发人员构建服务效率;通过路由管理器实现了服务框架中服务注册中心组件,为分布式系统提供服务发布、服务订阅、状态同步、消息通知等功能,并采用SOME/IP作为底层通信协议,为系统提供发布订阅式和请求响应式的服务调用方式。

基于流计算和大数据平台的实时交通流预测

目前交通流预测实时性差,很难满足在线分析和预测任务的需求,基于此提出一种Flink流计算框架和大数据平台结合的实时交通流预测方法。基于流计算框架实时捕捉和预处理数据,包括采用Flink的transform算子对数据进行校验和处理,将处理后的数据sink到大数据的HDFS文件系统,交由下一步的大数据并行框架进行分析建模与训练,实现基于流计算和大数据平台的实时交通流预测。实验结果表明,Flink能够实时捕捉和预处理交通流数据,把数据准时无误送入分布式文件系统中,在此基础上借助大数据框架下的并行分析和建模优势,在实时性数据分析与预测方面取得了较好的效果。

Survey of Distributed Computing Frameworks for Supporting Big Data Analysis

Distributed computing frameworks are the fundamental component of distributed computing systems.They provide an essential way to support the efficient processing of big data on clusters or cloud.The size of big data increases at a pace that is faster than the increase in the big data processing capacity of clusters.Thus,distributed computing frameworks based on the MapReduce computing model are not adequate to support big data analysis tasks which often require running complex analytical algorithms on extremely big data sets in terabytes.In performing such tasks,these frameworks face three challenges:computational inefficiency due to high I/O and communication costs,non-scalability to big data due to memory limit,and limited analytical algorithms because many serial algorithms cannot be implemented in the MapReduce programming model.New distributed computing frameworks need to be developed to conquer these challenges.In this paper,we review MapReduce-type distributed computing frameworks that are currently used in handling big data and discuss their problems when conducting big data analysis.In addition,we present a non-MapReduce distributed computing framework that has the potential to overcome big data analysis challenges.