Data Utilization-Based Adaptive Data Management Method for Distributed Storage System in WAN Environment

Recently,research on a distributed storage system that efficiently manages a large amount of data has been actively conducted following data production and demand increase.Physical expansion limits exist for traditional standalone storage systems,such as I/O and file system capacity.However,the existing distributed storage system does not consider where data is consumed and is more focused on data dissemination and optimizing the lookup cost of data location.And this leads to system performance degradation due to low locality occurring in a Wide Area Network(WAN)environment with high network latency.This problem hinders deploying distributed storage systems to multiple data centers over WAN.It lowers the scalability of distributed storage systems to accommodate data storage needs.This paper proposes a method for distributing data in a WAN environment considering network latency and data locality to solve this problem and increase overall system performance.The proposed distributed storage method monitors data utilization and locality to classify data temperature as hot,warm,and cold.With assigned data temperature,the proposed algorithm adaptively selects the appropriate data center and places data accordingly to overcome the excess latency from the WAN environment,leading to overall system performance degradation.This paper also conducts simulations to evaluate the proposed and existing distributed storage methods.The result shows that our proposed method reduced latency by 38%compared to the existing method.Therefore,the proposed method in this paper can be used in large-scale distributed storage systems over a WAN environment to improve latency and performance compared to existing methods,such as consistent hashing.

Tree-Structured Parallel Regeneration for Multiple Data Losses in Distributed Storage Systems Based on Erasure Codes

To reduce the time required to complete the regeneration process of erasure codes, we propose a Tree-structured Parallel Regeneration (TPR) scheme for multiple data losses in distributed storage systems. Under the scheme, two algorithms are proposed for the construction of multiple regeneration trees, namely the edge-disjoint algorithm and edge-sharing algorithm. The edge-disjoint algorithm constructs multiple independent trees, and is simple and appropriate for environments where newcomers and their providers are distributed over a large area and have few intersections. The edge-sharing algorithm constructs multiple trees that compete to utilize the bandwidth, and make a better utilization of the bandwidth, although it needs to measure the available band-width and deal with the bandwidth changes; it is therefore difficult to implement in practical systems. The parallel regeneration for multiple data losses of TPR primarily includes two optimizations: firstly, transferring the data through the bandwidth optimized-paths in a pipe-line manner; secondly, executing data regeneration over multiple trees in parallel. To evaluate the proposal, we implement an event-based simulator and make a detailed comparison with some popular regeneration methods. The quantitative comparison results show that the use of TPR employing either the edge-disjoint algorithm or edge-sharing algorithm reduces the regeneration time significantly.

Architecture Design of Global Distributed Storage System for Data Grid

This paper describes the architecture of global distributed storage system for data grid. It focue on the management and the capability for the maximum users and maximum resources on the Internet, as well as performance and other issues.

DESIGN OF EXACT REGENERATING HIERARCHICAL CODE FOR DISTRIBUTED STORAGE SYSTEM

Erasure code is widely used as the redundancy scheme in distributed storage system. When a storage node fails, the repair process often requires to transfer a large amount of data. Regenerating code and hierarchical code are two classes of codes proposed to reduce the repair bandwidth cost. Regenerating codes reduce the amount of data transferred by each helping node, while hierarchical codes reduce the number of nodes participating in the repair process. In this paper, we propose a "sub-code nesting framework" to combine them together. The resulting regenerating hierarchical code has low repair degree as hierarchical code and lower repair cost than hierarchical code. Our code can achieve exact regeneration of the failed node, and has the additional property of low updating complexity.

The Reliability Design and Availability Analysis of a Distributed Storage System

Current distributed parallel file systems and database systems can not satisfy the demands of data-intensive applications, such as storage capacity, access performance, reliability, scalability, and so on. Cluster-based storage sys tems have some shortcomings, too. To solve this kind of problems, a novel PC storage cluster solution is proposed, a distributed storage system based on 3-tiered agent architecture is designed, the system reliability model based on the masterslave backup mode is built, and the system availability is analyzed with the Markov model. According to the system availability formula and the values of the system parameters, the novel system can provide higher reliability and availability to satisfy users＇ requirements,

Grid Side Distributed Energy Storage Cloud Group End Region Hierarchical Time-Sharing Configuration Algorithm Based onMulti-Scale and Multi Feature Convolution Neural Network

There is instability in the distributed energy storage cloud group end region on the power grid side.In order to avoid large-scale fluctuating charging and discharging in the power grid environment and make the capacitor components showa continuous and stable charging and discharging state,a hierarchical time-sharing configuration algorithm of distributed energy storage cloud group end region on the power grid side based on multi-scale and multi feature convolution neural network is proposed.Firstly,a voltage stability analysis model based onmulti-scale and multi feature convolution neural network is constructed,and the multi-scale and multi feature convolution neural network is optimized based on Self-OrganizingMaps(SOM)algorithm to analyze the voltage stability of the cloud group end region of distributed energy storage on the grid side under the framework of credibility.According to the optimal scheduling objectives and network size,the distributed robust optimal configuration control model is solved under the framework of coordinated optimal scheduling at multiple time scales;Finally,the time series characteristics of regional power grid load and distributed generation are analyzed.According to the regional hierarchical time-sharing configuration model of“cloud”,“group”and“end”layer,the grid side distributed energy storage cloud group end regional hierarchical time-sharing configuration algorithm is realized.The experimental results show that after applying this algorithm,the best grid side distributed energy storage configuration scheme can be determined,and the stability of grid side distributed energy storage cloud group end region layered timesharing configuration can be improved.

Distributed Secure Storage Scheme Based on Sharding Blockchain

Distributed storage can store data in multiple devices or servers to improve data security.However,in today’s explosive growth of network data,traditional distributed storage scheme is faced with some severe challenges such as insufficient performance,data tampering,and data lose.A distributed storage scheme based on blockchain has been proposed to improve security and efficiency of traditional distributed storage.Under this scheme,the following improvements have been made in this paper.This paper first analyzes the problems faced by distributed storage.Then proposed to build a new distributed storage blockchain scheme with sharding blockchain.The proposed scheme realizes the partitioning of the network and nodes by means of blockchain sharding technology,which can improve the efficiency of data verification between nodes.In addition,this paper uses polynomial commitment to construct a new verifiable secret share scheme called PolyVSS.This new scheme is one of the foundations for building our improved distributed storage blockchain scheme.Compared with the previous scheme,our new scheme does not require a trusted third party and has some new features such as homomorphic and batch opening.The security of VSS can be further improved.Experimental comparisons show that the proposed scheme significantly reduces storage and communication costs.

Security Analysis on a Remote Data Auditing Scheme for Distributed Storage Systems

Remote data auditing becomes critical to ensure the storage reliability in distributed cloud storage.Recently,Le et al proposed an efficient private data auditing scheme NC-Audit designed for regenerating codes,which claimed that NC-Audit can effectively realize privacy-preserving data auditing for distributed storage systems.However,our analysis shows that NC-Audit is not secure for that the adversarial cloud can forge some illegal blocks to cheat the auditor successfully with a high probability even without storing the user’s whole data,when the coding field is large enough.

SA-RSR:a read-optimal data recovery strategy for XOR-coded distributed storage systems

To ensure the reliability and availability of data,redundancy strategies are always required for distributed storage systems.Erasure coding,one of the representative redundancy strategies,has the advantage of low storage overhead,which facilitates its employment in distributed storage systems.Among the various erasure coding schemes,XOR-based erasure codes are becoming popular due to their high computing speed.When a single-node failure occurs in such coding schemes,a process called data recovery takes place to retrieve the failed node’s lost data from surviving nodes.However,data transmission during the data recovery process usually requires a considerable amount of time.Current research has focused mainly on reducing the amount of data needed for data recovery to reduce the time required for data transmission,but it has encountered problems such as significant complexity and local optima.In this paper,we propose a random search recovery algorithm,named SA-RSR,to speed up single-node failure recovery of XOR-based erasure codes.SA-RSR uses a simulated annealing technique to search for an optimal recovery solution that reads and transmits a minimum amount of data.In addition,this search process can be done in polynomial time.We evaluate SA-RSR with a variety of XOR-based erasure codes in simulations and in a real storage system,Ceph.Experimental results in Ceph show that SA-RSR reduces the amount of data required for recovery by up to 30.0%and improves the performance of data recovery by up to 20.36%compared to the conventional recovery method.

DCRL-KG: Distributed Multi-Modal Knowledge Graph Retrieval Platform Based on Collaborative Representation Learning

The knowledge graph with relational abundant information has been widely used as the basic data support for the retrieval platforms.Image and text descriptions added to the knowledge graph enrich the node information,which accounts for the advantage of the multi-modal knowledge graph.In the field of cross-modal retrieval platforms,multi-modal knowledge graphs can help to improve retrieval accuracy and efficiency because of the abundant relational infor-mation provided by knowledge graphs.The representation learning method is sig-nificant to the application of multi-modal knowledge graphs.This paper proposes a distributed collaborative vector retrieval platform(DCRL-KG)using the multi-modal knowledge graph VisualSem as the foundation to achieve efficient and high-precision multimodal data retrieval.Firstly,use distributed technology to classify and store the data in the knowledge graph to improve retrieval efficiency.Secondly,this paper uses BabelNet to expand the knowledge graph through multi-ple filtering processes and increase the diversification of information.Finally,this paper builds a variety of retrieval models to achieve the fusion of retrieval results through linear combination methods to achieve high-precision language retrieval and image retrieval.The paper uses sentence retrieval and image retrieval experi-ments to prove that the platform can optimize the storage structure of the multi-modal knowledge graph and have good performance in multi-modal space.

On Distributed Object Storage Architecture Based on Mimic Defense

With the advent of the era of big data,cloud computing,Internet of things,and other information industries continue to develop.There is an increasing amount of unstructured data such as pictures,audio,and video on the Internet.And the distributed object storage system has become the mainstream cloud storage solution.With the increasing number of distributed applications,data security in the distributed object storage system has become the focus.For the distributed object storage system,traditional defenses are means that fix discovered system vulnerabilities and backdoors by patching,or means to modify the corresponding structure and upgrade.However,these two kinds of means are hysteretic and hardly deal with unknown security threats.Based on mimic defense theory,this paper constructs the principle framework of the distributed object storage system and introduces the dynamic redundancy and heterogeneous function in the distributed object storage system architecture,which increases the attack cost,and greatly improves the security and availability of data.

Locally Minimum Storage Regenerating Codes in Distributed Cloud Storage Systems

In distributed cloud storage systems, inevitably there exist multiple node failures at the same time. The existing methods of regenerating codes, including minimum storage regenerating(MSR) codes and minimum bandwidth regenerating(MBR) codes, are mainly to repair one single or several failed nodes, unable to meet the repair need of distributed cloud storage systems. In this paper, we present locally minimum storage regenerating(LMSR) codes to recover multiple failed nodes at the same time. Specifically, the nodes in distributed cloud storage systems are divided into multiple local groups, and in each local group(4, 2) or(5, 3) MSR codes are constructed. Moreover, the grouping method of storage nodes and the repairing process of failed nodes in local groups are studied. Theoretical analysis shows that LMSR codes can achieve the same storage overhead as MSR codes. Furthermore, we verify by means of simulation that, compared with MSR codes, LMSR codes can reduce the repair bandwidth and disk I/O overhead effectively.

Distributed energy storage node controller and control strategy based on energy storage cloud platform architecture

Based on the energy storage cloud platform architecture,this study considers the extensive configuration of energy storage devices and the future large-scale application of electric vehicles at the customer side to build a new mode of smart power consumption with a flexible interaction,smooth the peak/valley difference of the load side power,and improve energy efficiency.A plug and play device for customer-side energy storage and an internet-based energy storage cloud platform are developed herein to build a new intelligent power consumption mode with a flexible interaction suitable for ordinary customers.Based on the load perception of the power grid,this study aims to investigate the operating state and service life of distributed energy storage devices.By selecting an integrated optimal control scheme,this study designs a kind of energy optimization and deployment strategy for stratified partition to reduce the operating cost of the energy storage device on the client side.The effectiveness of the system and the control strategy is verified through the Suzhou client-side distributed energy storage demonstration project.

Robust and Trustworthy Data Sharing Framework Leveraging On-Chain and Off-Chain Collaboration

The proliferation of Internet of Things(IoT)systems has resulted in the generation of substantial data,presenting new challenges in reliable storage and trustworthy sharing.Conventional distributed storage systems are hindered by centralized management and lack traceability,while blockchain systems are limited by low capacity and high latency.To address these challenges,the present study investigates the reliable storage and trustworthy sharing of IoT data,and presents a novel system architecture that integrates on-chain and off-chain data manage systems.This architecture,integrating blockchain and distributed storage technologies,provides high-capacity,high-performance,traceable,and verifiable data storage and access.The on-chain system,built on Hyperledger Fabric,manages metadata,verification data,and permission information of the raw data.The off-chain system,implemented using IPFS Cluster,ensures the reliable storage and efficient access to massive files.A collaborative storage server is designed to integrate on-chain and off-chain operation interfaces,facilitating comprehensive data operations.We provide a unified access interface for user-friendly system interaction.Extensive testing validates the system’s reliability and stable performance.The proposed approach significantly enhances storage capacity compared to standalone blockchain systems.Rigorous reliability tests consistently yield positive outcomes.With average upload and download throughputs of roughly 20 and 30 MB/s,respectively,the system’s throughput surpasses the blockchain system by a factor of 4 to 18.

Data-driven Predictive Voltage Control for Distributed Energy Storage in Active Distribution Networks

Integration of distributed energy storage(DES)is beneficial for mitigating voltage fluctuations in highly distributed generator(DG)-penetrated active distribution networks(ADNs).Based on an accurate physical model of ADN,conventional model-based methods can realize optimal control of DES.However,absence of network parameters and complex operational states of ADN poses challenges to model-based methods.This paper proposes a data-driven predictive voltage control method for DES.First,considering time-series constraints,a data-driven predictive control model is formulated for DES by using measurement data.Then,a data-driven coordination method is proposed for DES and DGs in each area.Through boundary information interaction,voltage mitigation effects can be improved by interarea coordination control.Finally,control performance is tested on a modified IEEE 33-node test case.Case studies demonstrate that by fully utilizing multi-source data,the proposed predictive control method can effectively regulate DES and DGs to mitigate voltage violations.

Fuzzy Droop Control for SOC Balance and Stability Analysis of DC Microgrid with Distributed Energy Storage Systems

The unbalanced state of charge(SOC)of distributed energy storage systems(DESSs)in autonomous DC microgrid causes energy storage units(ESUs)to terminate operation due to overcharge or overdischarge,which severely affects the power quality.In this paper,a fuzzy droop control for SOC balance and stability analysis of DC microgrid with DESSs is proposed to achieve SOC balance in ESUs while maintaining a stable DC bus voltage.First,the charge and discharge modes of ESUs are determined based on the power supply requirements of the DC microgrid.One-dimensional fuzzy logic is then applied to establish the relationship between SOC and the droop coefficient R,in the aforementioned two modes.In addition,when integrated with voltage-current double closed-loop control,SOC balance in different ESUs is realized.To improve the balance speed and precision,an exponential acceleration factor is added to the input variable of the fuzzy controller.Finally,based on the average model of converter,the system-level stability of microgrid is analyzed.MATLAB/Simulink simulation results verify the effectiveness and rationality of the proposed method.

Low-Carbon Dispatching for Virtual Power Plant with Aggregated Distributed Energy Storage Considering Spatiotemporal Distribution of Cleanness Value

The scale of distributed energy resources is increasing,but imperfect business models and value transmission mechanisms lead to low utilization ratio and poor responsiveness.To address this issue,the concept of cleanness value of distributed energy storage(DES)is proposed,and the spatiotemporal distribution mechanism is discussed from the perspectives of electrical energy and cleanness.Based on this,an evaluation system for the environmental benefits of DES is constructed to balance the interests between the aggregator and the power system operator.Then,an optimal low-carbon dispatching for a virtual power plant(VPP)with aggregated DES is constructed,where-in energy value and cleanness value are both considered.To achieve the goal,a green attribute labeling method is used to establish a correlation constraint between the nodal carbon potential of the distribution network(DN)and DES behavior,but as a cost,it brings multiple nonlinear relationships.Subsequently,a solution method based on the convex envelope(CE)linear re-construction method is proposed for the multivariate nonlinear programming problem,thereby improving solution efficiency and feasibility.Finally,the simulation verification based on the IEEE 33-bus DN is conducted.The simulation results show that the multidimensional value recognition of DES motivates the willingness of resource users to respond.Meanwhile,resolving the impact of DES on the nodal carbon potential can effectively alleviate overcompensation of the cleanness value.

State-of-charge Balance Control and Safe Region Analysis for Distributed Energy Storage Systems with Constant Power Loads

This paper presents a fully distributed state-of-charge balance control (DSBC) strategy for a distributed energy storage system (DESS). In this framework, each energy storage unit (ESU) processes the state-of-charge (SoC) information from its neighbors locally and adjusts the virtual impedance of the droop controller in real-time to change the current sharing. It is shown that the SoC balance of all ESUs can be achieved. Due to virtual impedance, voltage deviation of the bus occurs inevitably and increases with load power. Meanwhile, widespread of the constant power load (CPL) in the power system may cause instability. To ensure reliable operation of DESS under the proposed DSBC, the concept of the safe region is put forward. Within the safe region, DESS is stable and voltage deviation is acceptable. The boundary conditions of the safe region are derived from the equivalent model of DESS, in which stability is analyzed in terms of modified Brayton-Moser's criterion. Both simulations and hardware experiments verify the accuracy of the safe region and effectiveness of the proposed DSBC strategy.

Enhancing Autonomy Capability in Regional Power Grids:A Strategic Planning Approach with Multiple Autonomous Evaluation Indexes

After the integration of large-scale DistributedGeneration(DG)into the distribution network,the randomness and volatility of its output result in a reduction of spatiotemporal alignment between power generation and demand in the distribution network,exacerbating the phenomenon of wind and solar power wastage.As a novel power system model,the fundamental concept of Regional Autonomous Power Grids(RAPGs)is to achieve localized management and energy autonomy,thereby facilitating the effective consumption of DGs.Therefore,this paper proposes a distributed resource planning strategy that enhances the autonomy capabilities of regional power grids by considering multiple evaluation indexes for autonomy.First,a regional Energy Storage(ES)configuration strategy is proposed.This strategy can select a suitable reference value for the upper limit of ES configuration based on the regional load andDGoutput to maximize the elimination of source load deviations in the region as the upper limit constraint of ES capacity.Then,a control strategy for regional ES is proposed,the charging and discharging reference line of ES is set,and multiple autonomy and economic indexes are used as objective functions to select different proportions of ES to control the distributed resources of the regional power grid and establish evaluation indexes of the internal regional generation and load power ratio,the proportion of power supply matching hours,new energy consumption rate and tie line power imbalance outside the region to evaluate changes in the regional autonomy capabilities.The final simulation results showthat in the real regional grid example,the planning method in the planning year in the region of the overall power supply matching hour ratio and new energy consumption rate increased by 3.9%and 4.8%on average,and the power imbalance of the tie line decreased by 7.8%on average.The proposed planning approach enables the maximization of regional autonomy while effectively smoothing the fluctuation of power exchange between the regional grid and the higher-level grid.This presents a rational and effective planning solution for the regional grid,facilitating the coordinated development between the region and the distribution network.

Massive Files Prefetching Model Based on LSTM Neural Network with Cache Transaction Strategy

In distributed storage systems,file access efficiency has an important impact on the real-time nature of information forensics.As a popular approach to improve file accessing efficiency,prefetching model can fetches data before it is needed according to the file access pattern,which can reduce the I/O waiting time and increase the system concurrency.However,prefetching model needs to mine the degree of association between files to ensure the accuracy of prefetching.In the massive small file situation,the sheer volume of files poses a challenge to the efficiency and accuracy of relevance mining.In this paper,we propose a massive files prefetching model based on LSTM neural network with cache transaction strategy to improve file access efficiency.Firstly,we propose a file clustering algorithm based on temporal locality and spatial locality to reduce the computational complexity.Secondly,we propose a definition of cache transaction according to files occurrence in cache instead of time-offset distance based methods to extract file block feature accurately.Lastly,we innovatively propose a file access prediction algorithm based on LSTM neural network which predict the file that have high possibility to be accessed.Experiments show that compared with the traditional LRU and the plain grouping methods,the proposed model notably increase the cache hit rate and effectively reduces the I/O wait time.