Hybrid Scalable Researcher Recommendation System Using Azure Data Lake Analytics

This research paper has provided the methodology and design for implementing the hybrid author recommender system using Azure Data Lake Analytics and Power BI. It offers a recommendation for the top 1000 Authors of computer science in different fields of study. The technique used in this paper is handling the inadequate Information for citation;it removes the problem of cold start, which is encountered by very many other recommender systems. In this paper, abstracts, the titles, and the Microsoft academic graphs have been used in coming up with the recommendation list for every document, which is used to combine the content-based approaches and the co-citations. Prioritization and the blending of every technique have been allowed by the tuning system parameters, allowing for the authority in results of recommendation versus the paper novelty. In the end, we do observe that there is a direct correlation between the similarity rankings that have been produced by the system and the scores of the participant. The results coming from the associated scrips of analysis and the user survey have been made available through the recommendation system. Managers must gain the required expertise to fully utilize the benefits that come with business intelligence systems [1]. Data mining has become an important tool for managers that provides insights about their daily operations and leverage the information provided by decision support systems to improve customer relationships [2]. Additionally, managers require business intelligence systems that can rank the output in the order of priority. Ranking algorithm can replace the traditional data mining algorithms that will be discussed in-depth in the literature review [3].

Realising Data-Centric Scientific Workflows with Provenance-Capturing on Data Lakes

Since their introduction by James Dixon in 2010,data lakes get more and more attention,driven by the promise of high reusability of the stored data due to the schema-on-read semantics.Building on this idea,several additional requirements were discussed in literature to improve the general usability of the concept,like a central metadata catalog including all provenance information,an overarching data governance,or the integration with(high-performance)processing capabilities.Although the necessity for a logical and a physical organisation of data lakes in order to meet those requirements is widely recognized,no concrete guidelines are yet provided.The most common architecture implementing this conceptual organisation is the zone architecture,where data is assigned to a certain zone depending on the degree of processing.This paper discusses how FAIR Digital Objects can be used in a novel approach to organize a data lake based on data types instead of zones,how they can be used to abstract the physical implementation,and how they empower generic and portable processing capabilities based on a provenance-based approach.

Wetland vegetation biomass estimation and mapping from Landsat ETM data: a case study of Poyang Lake

Poyang Lake is the largest freshwater lake in China. This paper conducted a digital and rapid investigation of the lake’s wetland vegetation biomass using Landsat ETM data acquired on April 16, 2000. First, utilizing the false color composite derived from the ETM data as one of the main references, the authors designed a reasonable sampling route for field measurement of the biomass, and carried it out on April 18–28, 2000. Then after both the sampling data and the ETM data were geometrically corrected to an equal-area projection of Albers, linear relationships among the sampling data and some transformed data derived from the ETM data and the ETM 4 were calculated. The results show that the sampling data is best relative to the band 4 data with a high correlation coefficient of 0.86, followed by the DVI and NDVI data with 0.83 and 0.80 respectively. Therefore, a linear regression model, which was based on the field data and band 4 data, was used to estimate the total biomass of entire Poyang Lake, and then the map of the biomass distribution was compiled.

Mapping of moraine dammed glacial lakes and assessment of their areal changes in the central and eastern Himalayas using satellite data

The relatively rapid recession of glaciers in the Himalayas and formation of moraine dammed glacial lakes(MDGLs) in the recent past have increased the risk of glacier lake outburst floods(GLOF) in the countries of Nepal and Bhutan and in the mountainous territory of Sikkim in India. As a product of climate change and global warming, such a risk has not only raised the level of threats to the habitation and infrastructure of the region, but has also contributed to the worsening of the balance of the unique ecosystem that exists in this domain that sustains several of the highest mountain peaks of the world. This study attempts to present an up to date mapping of the MDGLs in the central and eastern Himalayan regions using remote sensing data, with an objective to analyse their surface area variations with time from 1990 through 2015, disaggregated over six episodes. The study also includes the evaluation for susceptibility of MDGLs to GLOF with the least criteria decision analysis(LCDA). Forty two major MDGLs, each having a lake surface area greater than 0.2 km2, that were identified in the Himalayan ranges of Nepal, Bhutan, and Sikkim, have been categorized according to their surface area expansion rates in space and time. The lakes have been identified as located within the elevation range of 3800 m and6800 m above mean sea level(a msl). With a total surface area of 37.9 km2, these MDGLs as a whole were observed to have expanded by an astonishing 43.6% in area over the 25 year period of this study. A factor is introduced to numerically sort the lakes in terms of their relative yearly expansion rates, based on their interpretation of their surface area extents from satellite imageries. Verification of predicted GLOF events in the past using this factor with the limited field data as reported in literature indicates that the present analysis may be considered a sufficiently reliable and rapid technique for assessing the potential bursting susceptibility of the MDGLs. The analysis also indicates that, as of now, there are eight MDGLs in the region which appear to be in highly vulnerable states and have high chances in causing potential GLOF events anytime in the recent future.

Study on the Applicability of ERA5 Reanalysis Data at Lake Taihu

Lakes are an important component of the earth climate system. They play an important role in the study of basin weather forecasting, air quality forecasting, and regional climate research. The accuracy of driving variables is the basic premise to ensure the rationality of lake mode simulation. Based on the in-situ observations at Bifenggang site of the Lake Taihu Eddy flux Network from 2012 to 2017, this paper investigated temporal variations in temperature, relative humidity, wind speed, radiation components at different time scales (hourly, seasonal and interannual). ERA5 reanalysis data were compared with in-situ observation to quantify the error and evaluate the performance of reanalysis data. The results show that: 1) On the hourly scale, the ERA5 reanalysis data described air temperature, and downward long-wave radiation more accurately. 2) On the seasonal variation scale, the ERA5 reanalysis data described air temperature, and downward long-wave radiation more accurately. However, the descriptions of wind speed, relative humidity and downward short-wave have large deviations. 3) On the interannual scale, the ERA5 reanalysis data show a good performance for temperature, followed by downward longwave radiation, downward shortwave radiation and relative humidity.

基于数据中台的高校数据服务体系及快速应用构建研究

针对目前高校在数据集成、数据治理和数据服务等方面的问题,提出了基于数据中台的解决方案,在数据的采集和加工处理中引入数据湖、主题层和专题层,并对数据中台的整体架构设计、中台内数据的分层设计、数据模型的构建、数据服务体系的设计进行了详细阐述;对基于数据中台的快速应用构建,提出了采用大中台、微服务的系统建设方式,通过对系统进行解构,重新根据不同业务逻辑进行模块化组装,实现快速灵活构建业务系统。经过实际案例验证,该研究对于高校如何建设数据中台以及如何在中台上开展各类数字应用建设提供了很好的范例。

基于文献计量的盐湖研究进展及前沿态势分析

基于web of science和知网数据源,利用Derwent Data Analyzer,VOSvievwer和Incites对1960—2022年盐湖相关文献进行计量分析,以深入了解国际和国内盐湖研究进展和发展趋势。分析得出近60多年盐湖研究发文量整体上呈先稳定后增加的趋势;盐湖相关研究国际发文数量最高的期刊是International Journal of Systematic and Evolutionary Microbiology,国内发文量最高的期刊是《盐湖研究》。美国和中国是盐湖研究的核心力量,两者的总发文量、总引频次和高被引论文数均居前2位。盐湖研究领域发文量最高的机构是中国科学院,其隶属单位青海盐湖研究所发文量占比最高。国际文献作者前15位中8位来自中国。国际和国内研究热点演进分析得出今后盐湖资源的分离和提取将得到持续关注,高被引文章分析说明近年来盐湖研究关注最多的为盐湖资源的分离提取,尤其是锂资源。整体而言,有关盐湖研究还在不断拓展和延伸,未来我国应在文章质量、研究方向布局和影响力上继续提升,发挥国内盐湖研究的对外影响力。

基于集成学习的内陆水体叶绿素a浓度反演

利用卫星数据监测内陆或水质状态,对生态决策具有重要意义。基于具有高时空分辨率的哨兵二号卫星数据,联合2种集成学习算法反演山东省南四湖叶绿素a(Chla)浓度,结果表明经遥感反射率校正后的哨兵二号数据更加适用于水质反演。XGBoost模型在五折交叉验证反演结果上表现最优(R^(2)=0.732 5,RMSE=9.168 1μg/L),反演结果更符合实际情况。因此,使用该模型反演南四湖叶绿素a浓度,能较好地掌握其时空变化情况,对其他区域类似研究可提供一定参考。

Construction of lake bathymetry from MODIS satellite data and GIS from 2003 to 2011

In recent years, sedimentation conditions in Dongting Lake have varied greatly because of signifi cant changes in runoff and sediment load in the Changjiang(Yangtze) River following the construction of Three Gorges Dam. The topography of the lake bottom has changed rapidly because of the intense exchange of water and sediment between the lake and the Changjiang River. However, time series information on lake-bottom topographic change is lacking. In this study, we introduced a method that combines remote sensing data and in situ water level data to extract a record of Dongting Lake bottom topography from 2003 to 2011. Multi-temporal lake land/water boundaries were extracted from MODIS images using the linear spectral mixture model method. The elevation of water/land boundary points were calculated using water level data and spatial interpolation techniques. Digital elevation models of Dongting Lake bottom topography in different periods were then constructed with the multiple heighted waterlines. The mean root-mean-square error of the linear spectral mixture model was 0.036, and the mean predicted error for elevation interpolation was-0.19 m. Compared with fi eld measurement data and sediment load data, the method has proven to be most applicable. The results show that the topography of the bottom of Dongting Lake has exhibited uneven erosion and deposition in terms of time and space over the last nine years. Moreover, lake-bottom topography has undergone a slight erosion trend within this period, with 58.2% and 41.8% of the lake-bottom area being eroded and deposited, respectively.

铁路房建工程数据管控系统软件研发

针对铁路房建工程中存在人工汇总效率低、标准不统一、易出错等问题,开发一套铁路房建工程数据管控系统软件。该系统通过制定房建标准化分类命名及标准化的提资模板,实现数据的批量导入;同时通过输入相关参数,可以自动配置全线房屋并汇总房屋相关需求数据,各专业可以通过该系统开展具体的设计并将设计数据汇总至系统中,形成房建数据湖,并利用数据湖生成各专业的数据统计表格。该系统解决了日常工作中专业间提资数据不完整、不可信、获取慢等问题,并实现设计数据的即时共享,大幅提高设计效率及质量,促进铁路房建工程的数字化转型,有力支持以铁路全专业为主体的协同设计系统的研发。

云原生数据湖服务平台的设计与实现

云原生数据湖已经成为数据管理和分析领域的研究热点,相关技术和应用也得到了广泛的关注和探索。数据湖部署存在着成本高、组件之间兼容性差等问题,存算不分离制约着数据湖平台延展性,缺乏完备的数据入湖体系容易引起数据湖沼泽的形成,导致用户无法从中提取数据价值。该文设计并实现了云原生数据湖服务平台,平台以Kubernetes为底层构建云原生环境,结合容器技术将数据湖组件镜像化,同时设计数据湖存算分离方案来提高数据湖平台的可扩展性和可移植性,并配合监控、组装生产线将镜像容器化,实现数据湖上云操作。并建立用户入湖作业与云原生计算引擎之间的桥梁,对入湖信息进行预处理,提供多类型作业以满足多元化入湖场景,以统一catalog的方式将数据写入数据湖中。实际运行结果表明,该平台既提高了数据湖平台的灵活性和可靠性,又确保了元数据和数据资产的规范性存储。

基于游记大数据的环巢湖地区乡村旅游客源市场时空结构分析

以环巢湖地区作为研究对象,基于2016—2021年旅游网站的乡村旅游游记数据,结合统计分析以及ArcGIS空间分析法研究环巢湖地区乡村旅游客源市场的结构特征以及影响因素。结果表明:乡村旅游客源市场季节性明显,节假日及周末效应突出,其中4至10月为旅游旺季,11至3月为旅游淡季;乡村旅游客源市场的地理集中指数总体较高,客源呈集中分布特征,合肥市为主要客源地,客源吸引半径均值为219.59 km,属于近程客源市场,空间使用曲线是基本型曲线、“U”型曲线和Maxwell-Boltzman曲线3种曲线的复合,客源核心逐渐从长三角地区、京津冀地区向内陆地区、珠三角地区扩散。乡村旅游客源市场的时空结构是多种因素相互作用的结果,主要是由乡村旅游目的地的特殊性决定的,同时闲暇时间、气候舒适度、交通、客源地经济发展水平等也是重要影响因素。

铁路数据分布式湖仓一体架构分析与设计

科学合理的数据资源分类方法和行之有效的数据湖架构体系,可以支撑起铁路全业务数据的高效存储、组织和利用,并进一步支持并优化各项运营业务。文章首先对现有数据湖架构进行简要分析,确定选用湖仓一体的概念,将铁路数据以主题进行分类以适应业务处理需求;其次设计了铁路数据分布式湖仓一体架构,阐述了路局级子湖仓一体与国铁集团总湖仓一体的架构与功能,以及两者之间的数据流转过程;最后分析了所设计架构的特性与存在的问题,为进一步构建有效的铁路运营数据湖提供了参考。

面向水文数字化转型的数据治理研究

信息孤岛和业务割据是当前数字化转型中遇到的主要问题。为解决上述问题,本文以水文数据治理为研究对象,根据当前水文行业数据状况和发展方向,提出在DAMA数据治理框架下的水文数据治理思路;阐述元数据、主数据、数据集成、资源目录等治理工作的开展方法与策略;以长江委水文局数据治理系统为实例,形成适合水文行业的数据治理模式。研究成果可为水文部门信息化管理决策和外部数据应用提供支撑。

基于数据虚拟化模型的铁路客运站视频数据湖研究

针对当前铁路旅客服务与生产管控平台(简称:管控平台)智能视频分析应用中存在的视频接入方式不规范、智能分析模型精度低、数据传输带宽受限等问题,设计了视频数据接入及存储硬件架构,在带宽受限的情况下,实现了视频码流、设备管理信息的安全接入,满足视频数据存储应用需求;构建了基于数据虚拟化模型的铁路客运客运站视频数据湖体系架构,满足客运站智能视频分析的实时性需求;对设备配置能力进行估算,并对主要设备进行配置。文章的研究结果可为相关工程的建设、设计和实施提供参考。

2021-2022年鄱阳湖丰水期水体富营养化指标空间分布数据集

鄱阳湖是长江流域最大的通江湖泊,对维持长江流域生态功能具有举足轻重的作用。然而随着气候变化和人类活动的共同影响,其富营养化程度逐年增加,生态功能逐渐恶化。为研究鄱阳湖富营养化的时空变化特征,保护鄱阳湖水资源、水环境安全,于2021年和2022年夏季藻类生长高峰期基于野外采样规范均匀采集了鄱阳湖湖体72个点位的水样,在实验室严格按照国标和行业标准规定的方法完成了采样点水体富营养化指标总氮、总磷、高锰酸盐指数和叶绿素的测定,形成了鄱阳湖2021-2022年丰水期水体富营养化指标空间分布数据集。本数据集产品可准确表征鄱阳湖丰水期水体污染状况、富营养化状态及其在空间上的分布特征,对鄱阳湖流域污染管控,长江流域水环境保护具有重要的数据支撑意义。

1977—2021年博斯腾湖大湖区湖岸线动态变化

湖泊岸线的动态变化可以表征湖泊的变化。以博斯腾湖大湖区为研究区,基于1977—2021年的Landsat遥感影像数据和实测水位资料,利用MNDWI法提取水体信息,运用岸线发育系数与岸线强度,分析研究区岸线动态变化,探讨影响研究区岸线动态变化的主要因素。研究结果表明:1977—2021年研究区岸线变化最明显的区域主要是研究区北岸。1977—1989年北岸岸线向南迁移的最大距离为0.93 km;1989—2003年北岸岸线向北迁移的最大距离为5.12 km;2003—2013年北岸岸线向南迁移的最大距离为5.46 km;2013—2021年北岸岸线向北迁移的最大距离为5.11 km。1977—1989年研究区岸线长度以1.55 km/a的速率逐渐缩短;1989—2002年岸线以16.34 km/a的速率持续增长;2002—2012年岸线以22.67 km/a的速率明显缩短;2012—2021年岸线以14.36 km/a的速率快速增长。年降水量、年均气温以及开都河径流量等自然要素是研究区岸线、水位和水面面积变化的主要自然因素。研究区岸线长度和曲折性受水位和水面面积的影响较为明显,1977—2021年研究区水位、水面面积和岸线长度的变化趋势基本一致。

基于“湖仓一体”技术的城轨大数据平台设计与升级改造实践

为了探寻城市轨道交通行业大数据平台建设与升级改造的最优方案,本文以城轨大数据平台为研究对象,从城轨大数据平台发展历程出发,梳理城轨大数据平台发展的3个阶段,分析各阶段大数据平台所采用的技术与优缺点,重点总结当前阶段“湖仓一体”大数据技术所具备的湖仓一体、流批一体、OLTP+OLAP、多重负载等优点,研究了基于该技术的大数据平台架构升级改造设计要点,并将该技术在北京地铁数据中心的大数据平台升级改造中进行应用验证。结果表明:“湖仓一体”大数据平台技术兼具数据湖的低成本、数据仓库的高性能等优点,解决了原大数据平台在性能、容量与多用途支持上的不足,为城轨行业大数据平台建设与升级改造提供了新的解决思路。

基于数据湖的省级卫星直收站数据共享服务设计

针对省级卫星直收站数据存储分散造成共享应用不便的问题,提出卫星多源数据便捷高效访问和安全分级管理方案。引入数据湖技术,按照全国统一规范构建了基于气象大数据云平台的省级气象数据湖系统,基于此实现了FY-3、FY-4卫星地面直收站原始接收数据、加工处理产品等多种卫星数据入湖集中管理,并提供统一的数据共享服务,解决了海量多源异构数据的存储、管理和共享服务等技术难题。通过数据湖统一权限控制,实现对用户访问数据的分级管理,保障了数据应用的稳定性和安全性。通过数据湖统一文件目录服务,实现长时间序列文件的便捷快速获取,有效支撑卫星数据的服务应用和挖掘分析。

医院数据湖系统设计与应用

目的:基于大数据平台构建高兼容性数据湖,以降低数据存储成本,减少数据理解偏差,提高数据质量。方法:通过分析医院的实际业务和数据应用的需求,结合数据湖底座技术能力,完善医疗大数据平台,建立全量医疗数据入湖体系,实现海量数据的存储、汇聚、资源化利用和资产化。结果:医疗数据湖的建设和应用可助力数据治理,实现数据高效快速接入和处理,为医疗运营、科学研究、医疗质控、临床服务、决策管理等提供数据基础支持。结论:根据医院数据及应用的实际情况,提供便捷、高效的数据治理途径,提升医院的医疗能力,增强服务保障能力。