Research on the Application of Big Data and Cloud Computing Technology in Smart Campus

The current education field is experiencing an innovation driven by big data and cloud technologies,and these advanced technologies play a central role in the construction of smart campuses.Big data technology has a wide range of applications in student learning behavior analysis,teaching resource management,campus safety monitoring,and decision support,which improves the quality of education and management efficiency.Cloud computing technology supports the integration,distribution,and optimal use of educational resources through cloud resource sharing,virtual classrooms,intelligent campus management systems,and Infrastructure-as-a-Service(IaaS)models,which reduce costs and increase flexibility.This paper comprehensively discusses the practical application of big data and cloud computing technologies in smart campuses,showing how these technologies can contribute to the development of smart campuses,and laying the foundation for the future innovation of education models.

Big Data of Home Energy Management in Cloud Computing

A smart grid is the evolved form of the power grid with the integration of sensing,communication,computing,monitoring,and control technologies.These technologies make the power grid reliable,efficient,and economical.However,the smartness boosts the volume of data in the smart grid.To obligate full benefits,big data has attractive techniques to process and analyze smart grid data.This paper presents and simulates a framework to make sure the use of big data computing technique in the smart grid.The offered framework comprises of the following four layers:(i)Data source layer,(ii)Data transmission layer,(iii)Data storage and computing layer,and(iv)Data analysis layer.As a proof of concept,the framework is simulated by taking the dataset of three cities of the Pakistan region and by considering two cloud-based data centers.The results are analyzed by taking into account the following parameters:(i)Heavy load data center,(ii)The impact of peak hour,(iii)High network delay,and(iv)The low network delay.The presented framework may help the power grid to achieve reliability,sustainability,and cost-efficiency for both the users and service providers.

Big Data Storage Architecture Design in Cloud Computing

To solve the lag problem of the traditional storage technology in mass data storage and management,the application platform is designed and built for big data on Hadoop and data warehouse integration platform,which ensured the convenience for the management and usage of data.In order to break through the master node system bottlenecks,a storage system with better performance is designed through introduction of cloud computing technology,which adopts the design of master-slave distribution patterns by the network access according to the recent principle.Thus the burden of single access the master node is reduced.Also file block update strategy and fault recovery mechanism are provided to solve the management bottleneck problem of traditional storage system on the data update and fault recovery and offer feasible technical solutions to storage management for big data.

An overview of Hadoop applications in transportation big data

As an open-source cloud computing platform,Hadoop is extensively employed in a variety of sectors because of its high dependability,high scalability,and considerable benefits in processing and analyzing massive amounts of data.Consequently,to derive valuable insights from transportation big data,it is essential to leverage the Hadoop big data platform for analysis and mining.To summarize the latest research progress on the application of Hadoop to transportation big data,we conducted a comprehensive review of 98 relevant articles published from 2012 to the present.Firstly,a bibliometric analysis was performed using VOSviewer software to identify the evolution trend of keywords.Secondly,we introduced the core components of Hadoop.Subsequently,we systematically reviewed the98 articles,identified the latest research progress,and classified the main application scenarios of Hadoop and its optimization framework.Based on our analysis,we identified the research gaps and future work in this area.Our review of the available research highlights that Hadoop has played a significant role in transportation big data research over the past decade.Specifically,the focus has been on transportation infrastructure monitoring,taxi operation management,travel feature analysis,traffic flow prediction,transportation big data analysis platform,traffic event monitoring and status discrimination,license plate recognition,and the shortest path.Additionally,the optimization framework of Hadoop has been studied in two main areas:the optimization of the computational model of Hadoop and the optimization of Hadoop combined with Spark.Several research results have been achieved in the field of transportation big data.However,there is less systematic research on the core technology of Hadoop,and the breadth and depth of the integration development of Hadoop and transportation big data are not sufficient.In the future,it is suggested that Hadoop may be combined with other big data frameworks such as Storm and Flink that process real-time data sources to improve the real-time processing and analysis of transportation big data.Simultaneously,the research on multi-source heterogeneous transportation big data is still a key focus.Improving existing big data technology to enable the analysis and even data compression of transportation big data can lead to new breakthroughs for intelligent transportation.

Big Data with Cloud Computing:Discussions and Challenges

With the recent advancements in computer technologies,the amount of data available is increasing day by day.However,excessive amounts of data create great challenges for users.Meanwhile,cloud computing services provide a powerful environment to store large volumes of data.They eliminate various requirements,such as dedicated space and maintenance of expensive computer hardware and software.Handling big data is a time-consuming task that requires large computational clusters to ensure successful data storage and processing.In this work,the definition,classification,and characteristics of big data are discussed,along with various cloud services,such as Microsoft Azure,Google Cloud,Amazon Web Services,International Business Machine cloud,Hortonworks,and MapR.A comparative analysis of various cloud-based big data frameworks is also performed.Various research challenges are defined in terms of distributed database storage,data security,heterogeneity,and data visualization.

Big Data; Definition and Challenges

In recent years, due to the widespread use of electronic services and the use of social network as well, large volumes of information are being made that this information contains various types of things such as videos, photos, texts etc. besides large volume. Due to the high volume and the lack of specificity of this information, covering them through traditional and relational databases is not possible and modem solutions should be used for processing them, so that processing speed is also covered. Data storage for processing and the way of accessing to them in memory, network communication, covering required features for distributed system in solutions that are in use for storing big data, are the items that should be covered. In this paper, a collection of advantages and challenges of big data, special features and characteristics of them has been provided and with the introduction of technologies in use, storage methods are studied and research opportunities to continue the way will be introduced.

A Hadoop Performance Prediction Model Based on Random Forest

MapReduce is a programming model for processing large data sets, and Hadoop is the most popular open-source implementation of MapReduce. To achieve high performance, up to 190 Hadoop configuration parameters must be manually tunned. This is not only time-consuming but also error-pron. In this paper, we propose a new performance model based on random forest, a recently devel- oped machine-learning algorithm. The model, called RFMS, is used to predict the performance of a Hadoop system according to the system＇ s configuration parameters. RFMS is created from 2000 distinct fine-grained performance observations with different Hadoop configurations. We test RFMS against the measured performance of representative workloads from the Hadoop Micro-benchmark suite. The results show that the prediction accuracy of RFMS achieves 95% on average and up to 99%. This new, highly accurate prediction model can be used to automatically optimize the performance of Hadoop systems.

Analytics,challenges and applications in big data environment:a survey

Big data refer to the massive amounts and varieties of information in the structured and unstructured form,generated by social networking sites,biomedical equipment,financial companies,internet and websites,scientific sensors,agriculture engineering sources,and so on.This huge amount of data cannot be processed using traditional data processing systems and technologies.Big data analytics is a process of examining information and patterns from huge data.Hence,the process needs a system architecture for data collection,transmission,storage,processing and analysis,and visualization mechanisms.In this paper,we review the background and futuristic aspects of big data.We first introduce the history,background and related technologies of big data.We focus on big data system architecture,phases and classes of big data analytics.Then we present an open source big data framework to address some of the big data challenges.Finally,we discuss different applications of big data with some examples.

基于OpenStack云架构下Hadoop分布式集群部署实训教学过程研究

高职院校纷纷开设云计算技术和大数据技术专业,专业间既紧密联系又各有侧重,专业课程也有交叉,使学生在学习专业知识时存在一定误区和困惑。为解决教师教学实训过程中的瓶颈难点,深化学生对大数据和云计算技术的掌握程度,针对云计算技术中OpenStack项目和大数据技术中Hadoop项目进行融合部署,在OpenStack私有云成功配置的基础上,申请云中的资源创建云主机,进行Hadoop完全分布式集群部署和配置,使师生在该教学实训环节成功实现配置,进一步加深对大数据和云计算技术的理解,促进专业课程知识的深入学习。

Connected Geomatics in the big data era

The development of global informatization and its integration with industrialization symbolizes that human society has entered into the big data era.This article covers seven new characteristics of Geomatics(i.e.ubiquitous sensor,multi-dimensional dynamics,integration via networking,full automation in real time,from sensing to recognition,crowdsourcing and volunteered geographic information,and serviceoriented science),and puts forward the corresponding critical technical challenges in the construction of integrated space-air-ground geospatial networks.Through the discussions outlined in this paper,we propose a new development stage of Geomatics entitled‘Connected Geomatics,’which is defined as a multi-disciplinary science and technology that uses systematic approaches and integrates methods of spatio-temporal data acquisition,information extraction,network management,knowledge discovery,and spatial sensing and recognition,as well as intelligent location-based services pertaining to any physical objects and human activities on the earth.It is envisioned that the advancement of Geomatics will make a great contribution to human sustainable development.

基于Hadoop的大数据云计算处理的实现

随着大数据时代的到来,如何有效处理和分析海量数据成为亟待解决的问题。文章提出了一个基于Hadoop的大数据云计算处理方案,通过深入理解大数据的特性和云计算的基本架构,设计和实施了包括Hadoop集群环境搭建、数据存储、MapReduce任务设计在内的一整套处理流程。特别是对数据处理的优化和安全性问题,进行了详细的研究和实践。期望通过这种基于Hadoop的大数据云计算处理方案,能够有效应对大数据处理的挑战,提高数据处理的效率和安全性。同时也为未来在数据处理优化、数据安全性等方面的研究提供了参考和启示。

云计算中Hadoop技术研究与应用综述

Hadoop作为当今云计算与大数据时代背景下最热门的技术之一,其相关生态圈与Spark技术的结合一同影响着学术发展和商业模式。首先介绍了Hadoop的起源和优势,阐明相关技术原理,如MapReduce,HDFS,YARN,Spark等;然后着重分析了当前Hadoop学术研究成果,从MapReduce算法的改进与创新、HDFS技术的优化与创新、二次开发与其它技术相结合、应用领域创新与实践4个方面进行总结,并简述了国内外应用现状。而Hadoop与Spark结合是未来的趋势,最后展望了Hadoop未来研究的发展方向和亟需解决的问题。

基于Hadoop的智能电网数据安全存储设计

针对智能电网下海量数据的存储问题及数据保密性和完整性需求,分析了现有安全存储方案的特点,结合智能电网应用环境的特殊场合,设计了一种基于Hadoop的智能电网数据安全存储方案。该方案充分利用了HBase高性能优势和现代密码技术,将密钥与密文的管理分离,具有安全性好、密钥管理方便及效率高等特点。开发了基于Hadoop的原型系统,对方案的时间开销进行了分析,并通过相关实验证明了方案的有效性和可行性。

基于Hadoop分布式系统的地质环境大数据框架探讨

在分析目前地质环境数据现状基础上,结合大数据特点,分析了地质环境大数据特征,并结合目前地质环境应用情况,设计了地质环境大数据框架。该框架包括对地质环境大数据的数据进行清洗与转换、分布式数据存储管理、数据挖掘、文本检索、大数据可视化等功能。该框架对于后续实现地质环境大数据分析应用具有指导意义。

基于Hadoop的数据仓库构建模式研究

针对目前基于Hadoop的数据仓库一般采用"一对一"的模式或方法构建的情况,首先通过实例分析其不足之处;然后借鉴软件工程中的"生成器"设计模式的思想,提出一种Hadoop数据仓库的构建模式,称为"元数据驱动的生成器模式",用于构建基于Hadoop的数据仓库,即ETL过程。该模式具有两点优势:一是由元数据驱动,充分发挥了关系数据库管理系统对元数据操作的效率优势;二是识别了"通用知识"和"具体对象知识"两类知识,并在对知识的分类基础上设计和实现ETL过程,消除了"一对一"模式下大量不必要的重复操作。

云计算Hadoop平台的异常数据检测算法研究

近年来,随着我国互联网技术的飞速发展与大规模网络运算平台研究的深入,云平台下的数据处理已成为大规模数据的主要处理方式;但是,现有的云计算Hadoop平台在海量数据异常涌入状态下,常常出现数据逻辑错误、数据链完整性缺失、数据失效的问题,造成无法对上述异常数据进行有效检测处理,严重影响云计算Hadoop平台的数据运算准确性;针对上述问题,提出云计算Hadoop平台的异常数据检测算法研究方法;采用JNS数据采集筛查模组、算法逻辑补偿模组与动态反馈模组对现有的云端计算平台存在的问题进行针对性解决;通过仿真模拟实验证明,提出的云计算Hadoop平台的异常数据检测算法研究方法,具有异常数据识别率高,准确性高,速度快、可实施性强、稳定性好的特点。

高山无线发射台站智慧运维管理系统设计

自智慧广电战略提出以来,广播电视无线覆盖智慧化已成为广电行业技术研究的热点。本文详细介绍了高山无线发射台站的智慧化改造,通过在自台播出监控系统的基础上进行重新设计,构建了智慧运维管理系统。该系统实现了广播电视节目传输、发射的监控一体化云管理,并依托大数据、人工智能等技术,扩展了链路设备故障定位、智能诊断和播出预警等功能。这些改进措施全面提升了高山发射台站的安全播出保障能力。

云计算平台中分布式Hadoop数据挖掘关键技术研究（英文）

云计算环境下的大数据特征挖掘是大数据统计及分析的基础。为了提高聚类的准确度和速度,设计了一种基于分布式Hadoop平台和熵加权特征选择的数据挖掘方案。该方案首先采用无回路有向图对Hadoop平台下的Map Reduce作业流调度问题进行了分析。然后采用并行Map Reduce执行过程完成分布式计算。最后,采用熵加权聚类算法实现海量数据挖掘。仿真结果显示,提出的数据挖掘方案具有较好聚类效果和运行效率。

一种基于Hadoop云计算平台大数据聚类算法设计

传统的数据挖掘技术由于受到编程模型等的约束,产生了不同瓶颈,聚类算法的研究面临着海量的大数据处理与分析的挑战,新兴计算模型Hadoop作为一种可并行处理的云计算平台得到了广泛应用。文章对传统聚类挖掘算法进行改进和优化,在Hadoop云计算平台上进行K-means算法的并行化实现,降低算法的时间复杂度,提高了计算效率。实践证明,改进的K-means算法适合大规模数据集的聚类挖掘,具有高效、准确、稳定、安全等特性,适合于海量数据的分析和处理。

云计算的大数据分析技术在智能电网中的应用

探究了云计算的大数据分析技术在智能电网中的应用。通过分析智能电网所产生的数据量,介绍了云计算在数据存储、处理方面的优势。结合智能电网的需求,讨论了云计算大数据分析技术在智能电网运行优化、故障预测与管理、能源调度等方面的应用。最后,展望了云计算大数据分析技术在智能电网未来发展中的潜力。