【摘要】：近年來,隨著網(wǎng)絡(luò)傳輸技術(shù)的進(jìn)步與鏈路傳輸帶寬的提升,網(wǎng)絡(luò)流量激增,海量的網(wǎng)絡(luò)流量數(shù)據(jù)給網(wǎng)絡(luò)流量分析平臺(tái)帶來了許多存儲(chǔ)和計(jì)算方面的問題。Hadoop憑借其良好的容錯(cuò)性能,簡(jiǎn)單的并發(fā)編程模型已經(jīng)逐漸成為大數(shù)據(jù)處理平臺(tái)的首選,它也被廣泛的應(yīng)用在海量網(wǎng)絡(luò)流量分析應(yīng)用中。面對(duì)日益增長(zhǎng)的流量數(shù)據(jù),簡(jiǎn)單的對(duì)Hadoop集群進(jìn)行升級(jí)擴(kuò)容不僅會(huì)耗費(fèi)大量的人力物力,而且可能不會(huì)帶來集群性能的線性提升。因此,海量網(wǎng)絡(luò)流量分析平臺(tái)的作業(yè)調(diào)度及優(yōu)化工作就顯得尤為重要。本文首先對(duì)基于Hadoop的海量網(wǎng)絡(luò)流量分析平臺(tái)的體系架構(gòu)進(jìn)行了介紹。然后,通過與其他作業(yè)調(diào)度方法的對(duì)比,闡述了選用Oozie作為流量分析平臺(tái)的作業(yè)調(diào)度工具的原因,并展示了使用Oozie進(jìn)行作業(yè)調(diào)度的方法。接下來,在對(duì)網(wǎng)絡(luò)流量分析平臺(tái)典型Hadoop作業(yè)類型進(jìn)行總結(jié)的基礎(chǔ)上,針對(duì)不同類型的作業(yè)分別提出了不同的優(yōu)化方案,并對(duì)優(yōu)化的效果進(jìn)行了逐一的驗(yàn)證。最后,本文研究了采樣方法在網(wǎng)絡(luò)流量分析作業(yè)中的應(yīng)用。首先探究了采樣方法造成的流量分析過程中的相對(duì)誤差的影響因素,而后針對(duì)特定應(yīng)用場(chǎng)景提出了優(yōu)化的采樣策略,同時(shí)還指出了采樣方法在網(wǎng)絡(luò)流量分析應(yīng)用中的局限性。
[Abstract]:In recent years, with the progress of network transmission technology and the improvement of link transmission bandwidth, network traffic has increased dramatically. The massive network traffic data brings many problems in storage and computation to the network traffic analysis platform. With its good fault-tolerant performance, Hadoop has gradually become the first choice of big data processing platform because of its simple concurrent programming model. It is also widely used in mass network traffic analysis applications. In the face of increasing traffic data, simply upgrading and expanding the Hadoop cluster will not only consume a lot of manpower and material resources, but also may not lead to the linear improvement of cluster performance. Therefore, the task scheduling and optimization of mass network traffic analysis platform is particularly important. Firstly, the architecture of mass network traffic analysis platform based on Hadoop is introduced in this paper. Then, by comparing with other job scheduling methods, this paper expounds the reason why Oozie is chosen as the job scheduling tool of traffic analysis platform, and shows the method of job scheduling using Oozie. Then, on the basis of summarizing the typical Hadoop job types of network traffic analysis platform, different optimization schemes are proposed for different types of jobs, and the results of optimization are verified one by one. Finally, this paper studies the application of sampling method in network traffic analysis. This paper first explores the influence factors of the relative error in the flow analysis process caused by the sampling method, then puts forward an optimized sampling strategy for specific application scenarios, and also points out the limitations of the sampling method in the application of network traffic analysis.
【學(xué)位授予單位】：北京郵電大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2017
【分類號(hào)】：TP393.0

【相似文獻(xiàn)】

相關(guān)期刊論文 前10條

1 王宇;;網(wǎng)絡(luò)流量分析技術(shù)及其應(yīng)用[J];科技創(chuàng)業(yè)月刊;2010年03期

2 江萍萍;;網(wǎng)絡(luò)流量分析系統(tǒng)的設(shè)計(jì)研究[J];科技風(fēng);2012年19期

3 黃天戍,鄒俊峰,李俊娥,陳萍,劉s，

本文編號(hào)：2294116