期刊文献+
共找到7篇文章
< 1 >
每页显示 20 50 100
基于LZO的Hadoop文件归档优化方法
1
作者 张军 苏文浩 《计算机与现代化》 2023年第6期1-6,26,共7页
Hadoop分布式框架被广泛应用于各行业大数据处理领域,但在存储大量小文件时会产生更多的元数据信息,导致名称节点的内存占用过多,影响其对外提供高性能和高并发访问的能力。对小文件进行归档存储,是有效解决该问题的常用方案。同时,考... Hadoop分布式框架被广泛应用于各行业大数据处理领域,但在存储大量小文件时会产生更多的元数据信息,导致名称节点的内存占用过多,影响其对外提供高性能和高并发访问的能力。对小文件进行归档存储,是有效解决该问题的常用方案。同时,考虑到数据压缩可有效减少数据存储空间大小与网络数据传输负载,本文提出一种基于实时无损压缩算法LZO的Hadoop文件归档优化方法LA(LZO-Archive)。该方法先对小文件进行归档合并,在索引文件生成阶段融合LZO压缩算法,减少索引文件的生成时间。同时,设计一种文件压缩存储算法,对数据文件和索引文件进行压缩存储,有效减少了数据所占用的数据节点磁盘空间大小和名称节点内存空间占用大小。实验结果表明,相对于原始的HDFS数据存储方法、基准的文件归档方法HAR以及对比方法LHF,本文提出的LA方法在文件归档时间、名称节点内存占用、数据节点磁盘空间占用、文件访问时间等方面表现更好。 展开更多
关键词 HADOOP HDFS 小文件 归档方案 lzo算法
下载PDF
核聚变实验数据采集系统中数据压缩算法的研究 被引量:10
2
作者 朱琳 罗家融 田一海 《计算机工程》 CAS CSCD 北大核心 2003年第2期12-13,22,共3页
在分析当今普遍使用的压缩算法后,采用了适合核聚变实验装置数据采集和发布系统的LZO算法,并且给予具体应用,同时给出了性能评估,希望能对其它类似情况的数据压缩算法提供一定的参考价值。
关键词 核聚变实验 数据采集系统 数据压缩算法 超导托卡马克聚变装置
下载PDF
The development of data acquisition and processing application system for RF ion source 被引量:3
3
作者 Xiaodan ZHANG Xiaoying WANG +3 位作者 Chundong HU Caichao JIANG Yahong XlE Yuanzhe ZHAO 《Plasma Science and Technology》 SCIE EI CAS CSCD 2017年第7期124-129,共6页
As the key ion source component of nuclear fusion auxiliary heating devices, the radio frequency (RF) ion source is developed and applied gradually to offer a source plasma with the advantages of ease of control and... As the key ion source component of nuclear fusion auxiliary heating devices, the radio frequency (RF) ion source is developed and applied gradually to offer a source plasma with the advantages of ease of control and high reliability. In addition, it easily achieves long-pulse steady-state operation. During the process of the development and testing of the RF ion source, a lot of original experimental data will be generated. Therefore, it is necessary to develop a stable and reliable computer data acquisition and processing application system for realizing the functions of data acquisition, storage, access, and real-time monitoring. In this paper, the development of a data acquisition and processing application system for the RF ion source is presented. The hardware platform is based on the PXI system and the software is programmed on the LabVIEW development environment. The key technologies that are used for the implementation of this software programming mainly include the long-pulse data acquisition technology, multi- threading processing technology, transmission control communication protocol, and the Lempel-Ziv-Oberhumer data compression algorithm. Now, this design has been tested and applied on the RF ion source. The test results show that it can work reliably and steadily. With the help of this design, the stable plasma discharge data of the RF ion source are collected, stored, accessed, and monitored in real-time. It is shown that it has a very practical application significance for the RF experiments. 展开更多
关键词 RF ion source data acquisition data processing TCP lzo algorithm
下载PDF
Development of a Data Acquisition Control System for the First NBI on EAST 被引量:2
4
作者 张小丹 胡纯栋 +2 位作者 盛鹏 刘智民 赵远哲 《Plasma Science and Technology》 SCIE EI CAS CSCD 2013年第12期1254-1258,共5页
Neutral beam injection (NBI) heating is one of the most efficient auxiliary plasma heating methods for fusion devices. The data acquisition control system (DACS) with PXI (pe- ripheral component interconnect exte... Neutral beam injection (NBI) heating is one of the most efficient auxiliary plasma heating methods for fusion devices. The data acquisition control system (DACS) with PXI (pe- ripheral component interconnect extensions for instrumentation) data acquisition card for the first NBI system in the experimental advanced superconducting tokamak (EAST) is presented in this paper. As an important sub-system, DACS is designed to obtain physical measurement signals in the EAST NBI system and to deal and store these data with the Lempel-Ziv-Oberhumer (LZO) lossless data compression algorithm, as well as offer convenient data call-back and access inter- faces to the user for examining and analyzing the data. Experimental results show that accurate data will ensure that researchers correctly analyze it and then properly adjust the experimental parameters or operation, so DACS should take a large step in improving experimental efficiency. Tile hardware and software sections are briefly presented in this paper, and now this system has been tested to be able to work reliably and steadily. 展开更多
关键词 EAST NBI data acquisition TCP lzo algorithm
下载PDF
一种快速解压的无损压缩算法 被引量:4
5
作者 王松 房利国 +1 位作者 韩炼冰 刘鸿博 《通信技术》 2020年第5期1121-1126,共6页
随着信息技术的不断发展,大量数据给存储和传输都带来了巨大的挑战。数据压缩能够有效减少数据量,方便数据的处理和传输。无损压缩是一种利用数据的冗余特点进行压缩的压缩方法,解压时可以完全还原数据而不会有任何失真。在研究LZO算法... 随着信息技术的不断发展,大量数据给存储和传输都带来了巨大的挑战。数据压缩能够有效减少数据量,方便数据的处理和传输。无损压缩是一种利用数据的冗余特点进行压缩的压缩方法,解压时可以完全还原数据而不会有任何失真。在研究LZO算法的快速解压原理基础上,设计了一种新的压缩算法。该算法通过减少压缩数据中压缩块的数量,降低了解压程序的执行开销。测试结果表明,新算法可实现比LZO算法更快的解压速度。 展开更多
关键词 无损压缩 lzo 解压速度 压缩算法
下载PDF
The Implementation of Computer Data Processing Software for EAST NBI 被引量:1
6
作者 张小丹 胡纯栋 +3 位作者 盛鹏 赵远哲 吴德云 崔庆龙 《Plasma Science and Technology》 SCIE EI CAS CSCD 2014年第10期984-987,共4页
One of the most important project missions of neutral beam injectors is the implementation of 100 s neutral beam injection (NBI) with high power energy t.o the plasma of the EAST superconducting tokamak. Correspondi... One of the most important project missions of neutral beam injectors is the implementation of 100 s neutral beam injection (NBI) with high power energy t.o the plasma of the EAST superconducting tokamak. Correspondingly, it's necessary to construct a high-speed and reliable computer data processing system for processing experimental data, such as data acquisition, data compression and storage, data decompression and query, as well as data analysis. The implementation of computer data processing application software (CDPS) for EAST NBI is presented in this paper in terms of its functional structure and system realization. The set of software is programmed in C language and runs on Linux operating system based on TCP network protocol and multi-threading technology. The hardware mainly includes industrial control computer (IPC), data server, PXI DAQ cards and so on. Now this software has been applied to EAST NBI system, and experimental results show that the CDPS can serve EAST NBI very well. 展开更多
关键词 NBI data processing lzo algorithm TCP
下载PDF
基于EMS时间序列数据的实时全息无损压缩方法的研究与应用
7
作者 黄缙华 周伊琳 《广东电力》 2012年第9期82-86,共5页
针对时间序列数据库存在压缩实时性差、损失数据精度、压缩比低等缺陷,提出了时间序列数据实时全息无损压缩方法。介绍实时全息无损压缩方法利用差值、异或运算等方式实现时间序列数据的快速、无损的高效压缩过程,并在实验室测试中通过... 针对时间序列数据库存在压缩实时性差、损失数据精度、压缩比低等缺陷,提出了时间序列数据实时全息无损压缩方法。介绍实时全息无损压缩方法利用差值、异或运算等方式实现时间序列数据的快速、无损的高效压缩过程,并在实验室测试中通过实时全息无损压缩方法和Zip、Lzo压缩算法对120 MB的相同时间序列数据文件分别进行正弦数据、递增数据、功率扰动数据和开关量数据压缩测试。测试结果表明,实时全息无损压缩方法的压缩比最高可达到100∶1,可实现时间序列数据的无损、高效、快速压缩,具有强大的容错能力。 展开更多
关键词 数据采集与监视控制 时间序列数据 实时全息无损压缩方法 Zip压缩算法 lzo压缩算法 应用
下载PDF
上一页 1 下一页 到第
使用帮助 返回顶部