Cloud Computing as a disruptive technology, provides a dynamic, elastic and promising computing climate to tackle the challenges of big data processing and analytics. Hadoop and MapReduce are the widely used open sour...Cloud Computing as a disruptive technology, provides a dynamic, elastic and promising computing climate to tackle the challenges of big data processing and analytics. Hadoop and MapReduce are the widely used open source frameworks in Cloud Computing for storing and processing big data in the scalable fashion. Spark is the latest parallel computing engine working together with Hadoop that exceeds MapReduce performance via its in-memory computing and high level programming features. In this paper, we present our design and implementation of a productive, domain-specific big data analytics cloud platform on top of Hadoop and Spark. To increase user’s productivity, we created a variety of data processing templates to simplify the programming efforts. We have conducted experiments for its productivity and performance with a few basic but representative data processing algorithms in the petroleum industry. Geophysicists can use the platform to productively design and implement scalable seismic data processing algorithms without handling the details of data management and the complexity of parallelism. The Cloud platform generates a complete data processing application based on user’s kernel program and simple configurations, allocates resources and executes it in parallel on top of Spark and Hadoop.展开更多
The rapid growth of IP traffic has contributed to wide deployment of optical devices in elastic optical network.However,the passband shape of wavelength selective switches(WSSs)that are used in reconfigurable optical ...The rapid growth of IP traffic has contributed to wide deployment of optical devices in elastic optical network.However,the passband shape of wavelength selective switches(WSSs)that are used in reconfigurable optical add-drop multiplexer(ROADM)/optical cross connect(OXC)is not ideal,causing the narrowing of spectrum.Spectral narrowing will lead to signal impairment.Therefore,guard-bands need to be inserted between adjacent paths which will cause the waste of resources.In this paper,we propose a service-based intelligent aggregation node selection and area division(ANS-AD)algorithm.For the rationality of the aggregation node selection,the ANS-AD algorithm chooses the aggregation nodes according to historical traffic information based on big data analysis.Then the ANS-AD algorithm divides the topology into areas according to the result of the aggregation node selection.Based on the ANS-AD algorithm,we propose a time-domain and spectral-domain flow aggregation(TS-FA)algorithm.For the purpose of reducing resources'waste,the TS-FA algorithm attempts to reduce the insertion of guard-bands by time-domain and spectral-domain flow aggregation.Moreover,we design a time-domain and spectral-domain flow aggregation module on software defined optical network(SDON)architecture.Finally,a simulation is designed to evaluate the performance of the proposed algorithms and the results show that our proposed algorithms can effectively reduce the resource waste.展开更多
文摘Cloud Computing as a disruptive technology, provides a dynamic, elastic and promising computing climate to tackle the challenges of big data processing and analytics. Hadoop and MapReduce are the widely used open source frameworks in Cloud Computing for storing and processing big data in the scalable fashion. Spark is the latest parallel computing engine working together with Hadoop that exceeds MapReduce performance via its in-memory computing and high level programming features. In this paper, we present our design and implementation of a productive, domain-specific big data analytics cloud platform on top of Hadoop and Spark. To increase user’s productivity, we created a variety of data processing templates to simplify the programming efforts. We have conducted experiments for its productivity and performance with a few basic but representative data processing algorithms in the petroleum industry. Geophysicists can use the platform to productively design and implement scalable seismic data processing algorithms without handling the details of data management and the complexity of parallelism. The Cloud platform generates a complete data processing application based on user’s kernel program and simple configurations, allocates resources and executes it in parallel on top of Spark and Hadoop.
基金funded by ZTE Industry-Academia-Research Cooperation Funds under Grant No.2017110031005226
文摘The rapid growth of IP traffic has contributed to wide deployment of optical devices in elastic optical network.However,the passband shape of wavelength selective switches(WSSs)that are used in reconfigurable optical add-drop multiplexer(ROADM)/optical cross connect(OXC)is not ideal,causing the narrowing of spectrum.Spectral narrowing will lead to signal impairment.Therefore,guard-bands need to be inserted between adjacent paths which will cause the waste of resources.In this paper,we propose a service-based intelligent aggregation node selection and area division(ANS-AD)algorithm.For the rationality of the aggregation node selection,the ANS-AD algorithm chooses the aggregation nodes according to historical traffic information based on big data analysis.Then the ANS-AD algorithm divides the topology into areas according to the result of the aggregation node selection.Based on the ANS-AD algorithm,we propose a time-domain and spectral-domain flow aggregation(TS-FA)algorithm.For the purpose of reducing resources'waste,the TS-FA algorithm attempts to reduce the insertion of guard-bands by time-domain and spectral-domain flow aggregation.Moreover,we design a time-domain and spectral-domain flow aggregation module on software defined optical network(SDON)architecture.Finally,a simulation is designed to evaluate the performance of the proposed algorithms and the results show that our proposed algorithms can effectively reduce the resource waste.
