Parallel naive Bayes algorithm for large-scale Chinese text classification based on spark

The sharp increase of the amount of Internet Chinese text data has significantly prolonged the processing time of classification on these data.In order to solve this problem,this paper proposes and implements a parallel naive Bayes algorithm(PNBA)for Chinese text classification based on Spark,a parallel memory computing platform for big data.This algorithm has implemented parallel operation throughout the entire training and prediction process of naive Bayes classifier mainly by adopting the programming model of resilient distributed datasets(RDD).For comparison,a PNBA based on Hadoop is also implemented.The test results show that in the same computing environment and for the same text sets,the Spark PNBA is obviously superior to the Hadoop PNBA in terms of key indicators such as speedup ratio and scalability.Therefore,Spark-based parallel algorithms can better meet the requirement of large-scale Chinese text data mining.

An Experimental Comparative Study on Three Classification Algorithms

Classification algorithm is one of the key techniques to affect text automatic classification system’s performance, play an important role in automatic classification research area. This paper comparatively analyzed k-NN. VSM and hybrid classification algorithm presented by our research group. Some 2000 pieces of Internet news provided by ChinaInfoBank are used in the experiment. The result shows that the hybrid algorithm’s performance presented by the groups is superior to the other two algorithms.

A parallel algorithm for detecting traffic patterns using stay point features and moving features

In order to detect the traffic pattern of moving objects in the city more accurately and quickly, a parallel algorithm for detecting traffic patterns using stay points and moving features is proposed. First, the features of the stay points in different traffic patterns are extracted, that is, the stay points of various traffic patterns are identified, respectively, and the clustering algorithm is used to mine the unique features of the stop points to different traffic patterns. Then, the moving features in different traffic patterns are extracted from a trajectory of a moving object, including the maximum speed, the average speed, and the stopping rate. A classifier is constructed to predict the traffic pattern of the trajectory using the stay points and moving features. Finally, a parallel algorithm based on Spark is proposed to detect traffic patterns. Experimental results show that the stay points and moving features can reflect the difference between different traffic modes to a greater extent, and the detection accuracy is higher than those of other methods. In addition, the parallel algorithm can increase the speed of identifying traffic patterns.

A Study of Parallelism in English Speeches

Every public speaker prepares his or her public speech meticulously.Witty remarks emerge in an endless stream,and demonstrate the rhetoric beauty of English to a great extent.Almost every speaker employs parallelism in his or her public speeches.The present paper is intended to study the concept,the classification and the significance of parallelism in English.

Efficient Parallel Processing of k-Nearest Neighbor Queries by Using a Centroid-based and Hierarchical Clustering Algorithm

The k-Nearest Neighbor method is one of the most popular techniques for both classification and regression purposes.Because of its operation,the application of this classification may be limited to problems with a certain number of instances,particularly,when run time is a consideration.However,the classification of large amounts of data has become a fundamental task in many real-world applications.It is logical to scale the k-Nearest Neighbor method to large scale datasets.This paper proposes a new k-Nearest Neighbor classification method(KNN-CCL)which uses a parallel centroid-based and hierarchical clustering algorithm to separate the sample of training dataset into multiple parts.The introduced clustering algorithm uses four stages of successive refinements and generates high quality clusters.The k-Nearest Neighbor approach subsequently makes use of them to predict the test datasets.Finally,sets of experiments are conducted on the UCI datasets.The experimental results confirm that the proposed k-Nearest Neighbor classification method performs well with regard to classification accuracy and performance.

PT-MIL:Parallel transformer based on multi-instance learning for osteoporosis detection in panoramic oral radiography

Osteoporosis is a systemic disease characterized by low bone mass,impaired bone microstruc-ture,increased bone fragility,and a higher risk of fractures.It commonly affects postmenopausal women and the elderly.Orthopantomography,also known as panoramic radiography,is a widely used imaging technique in dental examinations due to its low cost and easy accessibility.Previous studies have shown that the mandibular cortical index(MCI)derived from orthopantomography can serve as an important indicator of osteoporosis risk.To address this,this study proposes a parallel Transformer network based on multiple instance learning.By introducing parallel modules that alleviate optimization issues and integrating multiple-instance learning with the Transformer architecture,our model effectively extracts information from image patches.Our model achieves an accuracy of 86%and an AUC score of 0.963 on an osteoporosis dataset,which demonstrates its promising and competitive performance.

基于大数据挖掘的城市规划异构数据调度平台

为了提高城市规划异构数据资源的调度效果,设计基于大数据挖掘的城市规划异构数据调度平台。通过配置FPGA采集异构大数据;采用大数据挖掘技术中的二进制分裂聚类算法分类存储数据;基于粒子群聚类算法分发匹配调度任务;基于大数据处理技术优化数据调度效率,完成城市规划异构数据资源调度。测试结果显示:该平台的数据采集速度均在6800字节/s以上;弗里德曼检验和调整随机误差结果最高值分别为0.965和0.02,数据分类效果良好;数据调度匹配率均在0.94以上,提高了城市规划异构数据资源调度效果。

基于并联型神经网络的环境声音分类

针对传统单输入模型在环境声音分类中准确率不高的问题,提出一种基于时域特征和频域特征并联型特征融合神经网络。在该网络中,首先通过数据增强的方法来处理原始音频;其次处理后的原始音频数据和梅尔(Mel)频谱特征数据分别送入原始波形网络和Mel频谱网络,得到其时域和频谱特征后,进行特征融合;最后,将特征融合后的结果送入SoftMax分类器进行分类。本文在UrbanSound8K数据集上进行了实验验证,最终分类准确率高达96.03%,优于其他模型。

SFA-ConvNeXt:逐级聚合多尺度ConvNeXt的皮肤镜图像分类

皮肤癌的早期发现对患者的五年生存率有着显著的提高,然而由于早期恶性肿瘤在皮肤中的病变非常细微,其症状并不明显,专业医生需要进行多次活检并提取病变组织才可以诊断出病变类型。现有的机器学习方法由于难以同时关注空间细节信息与浅层语义特征,其在皮肤病变图像中识别准确率并不高。为了有效表示空间位置和浅层特征信息,避免模型过于关注细节信息导致易分图像误分类等问题,提出了一种基于ConvNeXt的逐级聚合注意力网络。该方法通过分层ConvNeXt编码器逐层提取病变区域的深层和浅层特征,并通过并行空间注意力有效整合空间位置信息和深层或浅层语义特征,聚合多尺度上下文信息。同时,设计逐级特征聚合模块有效整合深层和浅层特征,并通过动态调整权重的方式将深层和浅层特征聚合,高度符合专业医生对皮肤镜图像分类时粗略观察和细微观察的过程。在ISIC2018、ISIC2019数据集中上进行实验测试,其准确率、精确率、召回率和F1-Score分别是95.27%、93.76%、92.83%、93.18%与92.63%、91.06%、87.05%、88.81%。通过与ConvNeXt相比,准确率分别提升了2.13和3.29个百分点,证明其能有效地提取细节特征和粗略特征,为皮肤镜图像的诊断提供新的依据。

基于多模态数据融合的飞行员注视区域分类

为了解决图像采集过程中眼图消失和头部姿态估计不准确的问题,利用基于非接触式的眼部信息获取方法采集人脸图像,从单个图像帧中确定飞行员当前的注视方向.同时,针对现有网络忽略头部运动对视线造成遮挡所导致的分类效果不佳问题,结合人脸图像与头部姿态特征,通过改进的MobileViT模型提出一种用于飞行员注视区域分类的多模态数据融合网络.首先提出了多模态数据融合模块解决特征拼接过程中尺寸不平衡导致的过拟合问题,其次提出一种基于并行分支SE机制的逆残差块,充分利用网络浅层的空间和通道特征信息,并结合Transformer的全局注意力机制捕捉多尺度特征.最后,重新设计了Mobile Block结构,使用深度可分离卷积降低模型复杂度.利用自制数据集FlyGaze对新模型和主流基线模型进行对比,实验结果表明, PilotT模型对注视区域0、3、4、5的分类准确率均在92%以上,且对人脸发生偏转的情况具有较强适应力.研究结果对提升飞行训练质量以及飞行员意图识别和疲劳评估具有实际应用价值.

ParSymG: a parallel clustering approach for unsupervised classification of remotely sensed imagery

Symmetry is a common feature in the real world.It may be used to improve a classification by using the point symmetry-based distance as a measure of clustering.However,it is time consuming to calculate the point symmetry-based distance.Although an efficient parallel point symmetry-based K-means algorithm(ParSym)has been propsed to overcome this limitation,ParSym may get stuck in sub-optimal solutions due to the K-means technique it used.In this study,we proposed a novel parallel point symmetry-based genetic clustering(ParSymG)algorithm for unsupervised classification.The genetic algorithm was introduced to overcome the sub-optimization problem caused by inappropriate selection of initial centroids in ParSym.A message passing interface(MPI)was used to implement the distributed master–slave paradigm.To make the algorithm more time-efficient,a three-phase speedup strategy was adopted for population initialization,image partition,and kd-tree structure-based nearest neighbor searching.The advantages of ParSymG over existing ParSym and parallel K-means(PKM)alogithms were demonstrated through case studies using three different types of remotely sensed images.Results in speedup and time gain proved the excellent scalability of the ParSymG algorithm.

Parallelizing maximum likelihood classification (MLC) for supervised image classification by pipelined thread approach through high-level synthesis (HLS) on FPGA cluster

High spectral,spatial,vertical and temporal resolution data are increasingly available and result in the serious challenge to pro-cess big remote-sensing images effectively and efficiently.This article introduced how to conduct supervised image classification by implementing maximum likelihood classification(MLC)over big image data on a field programmable gate array(FPGA)cloud.By comparing our prior work of implementing MLC on conventional cluster of multicore computers and graphics processing unit,it can be concluded that FPGAs can achieve the best performance in comparison to conventional CPU cluster and K40 GPU,and are more energy efficient.The proposed pipelined thread approach can be extended to other image-processing solutions to handle big data in the future.

功能对等理论视角下外宣文本排比修辞英译探究

该文基于功能对等理论,探讨外宣文本排比修辞的英译策略。首先,概述了功能对等理论,强调其追求多维度对等,而非单纯的形式对应;其次,对汉英排比修辞的定义、汉英排比修辞的分类及汉英排比修辞的差异进行了阐述,并分析外宣文本排比修辞的语言单位分类及其英译处理,包括直译与意译、增译与省译、重组与转换等翻译方法。总结指出,在翻译外宣文本排比修辞时,译者应依据语境,注重原文内容与意义的传达,灵活运用多种翻译技巧,以实现忠实性与可读性的统一。同时,遵循“内容优先”原则,促进中国文化的国际传播,进而提升国家形象与软实力。

面向信息系统推荐与决策的高阶张量分析方法

张量数据(或多维数组)在各个行业的信息系统中广泛存在,例如医疗系统中的功能性磁共振成像(fMRI)数据和商品数据信息系统中的用户-产品数据.将这些数据用以预测张量特征与单变量响应之间的关系,可以实现数据赋能,提供更精准的服务或解决方案,例如疾病决策诊断或商品推荐.然而,现有的张量回归方法存在2个主要问题:一是可能丢失了张量的空间信息,导致预测结果不准确;二是计算成本过高,导致服务或解决方案不及时.对于具有高阶结构的大规模数据而言,这2点则显得更为突出.因此为了实现数据赋能,即利用张量数据来提高信息服务或解决方案的质量和效率,提出了稀疏低秩张量回归模型(sparse and low-rank tensor regression model,SLTR).该模型通过对张量系数应用l1范数和张量核范数使得张量系数具有稀疏性和低秩性两大特点,这样既保留了张量的结构信息又可以方便地解释数据.利用近端梯度方法优化了混合正则化器,使得求解过程可扩展且高效.除此之外证明了SLTR的严格误差界.在多个模拟数据集和一个视频数据集上的实验结果表明,SLTR相比于之前的方法,在更短的时间内获得了更好的预测性能.

基于随机森林的电网GIS数据分布式存储方法

内部电网地理信息系统(Geographic Information Systern,GIS)数据体量增加,对电网数据存储性能造成了极大的困难,为此,提出一种基于随机森林的电网GIS数据分布式存储方法。以跨域资源共享(Cross-Origin Resource Sharing,CORS)技术在电网GIS空间信息服务平台中获取的电网GIS数据为基础,根据类区分度数值选择电网GIS数据特征,引入随机森林算法分类处理电网GIS数据,将其合理分发给不同的服务器,采用并行处理手段存储分类数据,从而实现了电网GIS数据的分布式存储。实验数据显示:应用所提方法后,电网GIS数据分类精度达到了96.8%,电网GIS数据分布式存储时间最小值为5.2 s,充分证实了所提方法数据存储性能更佳。

Parallelizing maximum likelihood classification on computer cluster and graphics processing unit for supervised image classification

Supervised image classification has been widely utilized in a variety of remote sensing applications.When large volume of satellite imagery data and aerial photos are increasingly available,high-performance image processing solutions are required to handle large scale of data.This paper introduces how maximum likelihood classification approach is parallelized for implementation on a computer cluster and a graphics processing unit to achieve high performance when processing big imagery data.The solution is scalable and satisfies the need of change detection,object identification,and exploratory analysis on large-scale high-resolution imagery data in remote sensing applications.

大数据下的机器学习算法综述

随着产业界数据量的爆炸式增长,大数据概念受到越来越多的关注.由于大数据的海量、复杂多样、变化快的特性,对于大数据环境下的应用问题,传统的在小数据上的机器学习算法很多已不再适用.因此,研究大数据环境下的机器学习算法成为学术界和产业界共同关注的话题.文中主要分析和总结当前用于处理大数据的机器学习算法的研究现状.此外,并行是处理大数据的主流方法,因此介绍一些并行算法,并引出大数据环境下机器学习研究所面临的问题.最后指出大数据机器学习的研究趋势.

KNN分类算法的MapReduce并行化实现

为了提高k-nearest neighbor algorithm(KNN)算法处理大数据集的能力,本文利用Map Reduce并行编程模型,同时结合KNN算法自身的特点,给出了KNN算法在Hadoop平台下的并行化实现。通过设计Map、Combine和Reduce 3个函数,实现了KNN算法的并行化。Map函数完成每个测试样本与训练样本之间的相似度计算,Combine函数作为一个本地的Reduce操作,用以减少中间计算量及通信开销,Reduce函数则根据上述函数得到的中间结果计算出k近邻并作出分类判断。实验结果表明:较之以往的单机版方法,在Hadoop集群上实现的并行化KNN算法具有较好的加速比和良好的扩展性。

荧光光谱及平行因子分析法在植物油鉴别中的应用

利用FS920荧光光谱仪测量市售的八种植物油（大豆油、玉米油、橄榄油、稻米油、花生油、核桃油、葵花籽油和芝麻油）共22个样品的荧光光谱，并对其数据矩阵（EEMs）进行平行因子分析，结合荧光谱分析的直观物质表征和平行因子法对灰色体系的组分识别优势，实现了植物油的种类区分与鉴别。综合分析植物油在特定范围内（激发波长为250～550 nm，发射波长为260～750 nm）的三维荧光光谱和等高线光谱图，给出了各植物油峰位、峰数和峰强等特征信息，确定了植物油各荧光谱峰相应的荧光物质（不饱和脂肪酸类、维生素E及其衍生物、叶绿素及类胡萝卜素）；将平行因子模型应用于植物油光谱数据矩阵的分析，确定了平行因子分析模型的因子数及各因子的物质基础（维生素E及其衍生物、亚油酸和亚麻酸、脂肪酸氧化产物、植物油氧化产物）。建立了植物油的4因子激发-发射光谱轮廓图和样品因子投影得分图。通过对植物油荧光光谱的图谱特征和其数据阵平行因子模型的分析，证实荧光光谱技术和平行因子分析法对植物油进行分析和种类鉴别的有效性。

基于机器视觉的机器人分拣系统的设计与实现

传统的分拣作业一般采用示教或离线编程方式,当工作环境发生改变时系统无法即时的作出相应的调整,针对传统分拣方式中存在的不足,以并联型机器人和康奈视In-Sight7000型工业智能相机为基础,搭建一个基于机器视觉的工业机器人分拣系统。该分拣系统结合了并联型机器人和机器视觉两方面的技术优势,对经典的Canny边缘提取算子做出了相应的改进,通过提取图象边缘信息作为匹配的特征,克服了光照变化对视觉系统造成的影响,实现了当分拣对象由传送带运送进入相机视野时,系统可以高速的对分拣对象进行识别和分拣工作。该系统显著提升了机器人对工作环境的适应能力,提高了生产效率和系统柔性。同时,实验结果证明了该系统软硬件设计正确,分拣成功率高。