基于Hadoop的校园网视频流量访问偏好分析

Hadoop based analysis of access preference for campus video traffic

针对传统视频用户访问日志的偏好分析方法存在数据客观性差和用户关联分析困难等问题,在传统偏好分析方法的基础上,面向互联网数据原始流量,提出一种基于主流大数据平台技术Hadoop的校园网视频用户访问偏好分析方案。该方案利用网络爬虫和深度包检测技术,对视频访问内容进行精细化识别,进而研究了校园网视频流量的访问偏好,并对比了My SQL和Hive的查询效率。结果表明,文化层次的差异导致了视频用户群体的不同需求,低成本硬件环境下对大数据的处理Hive更显健壮性。另外,该方案能稳定可靠地实现对校园网视频流量访问的偏好分析,捕捉用户网络舆情,制订定向营销方案并提供个性化视频推荐服务。经现网测试验证,设计的视频访问偏好分析方案达到了预期的效果。

There are poor objectivity of data and difficult association analysis in traditional user preference analysis approach with video access log. To solve this problem,the Hadoop based scheme of video user access preference analysis in campus network is proposed by using the original traffic from internet in this paper. The scheme was designed to refine identification of video access content by using the technology of web crawler and deep packet inspection. And the access preference for campus video traffic is analyzed in further. The query efficiency was compared between My SQL and Hive at the same time. The results demonstrate that the difference level of culture leads to the variation of video needs among user groups,and under the low cost hardware environment,Hive is robust for the processing of large data. The scheme is stable and reliable to realize the analysis of access preference for campus video traffic,capturing user network public opinions,working out customized marketing plans and providing service of personalized video recommendation. Through testing in current network environment,the scheme of video user preference analysis proposed in this paper works well as what is expected.