【摘要】：針對(duì)當(dāng)前基于軟/硬件的流導(dǎo)出技術(shù)存在的數(shù)據(jù)失真、不易擴(kuò)展等問題,本文提出一種準(zhǔn)確、通用、易擴(kuò)展的高速網(wǎng)絡(luò)流量處理框架NTCI-Flow。首先,基于PF_RING DNA實(shí)現(xiàn)了高性能的網(wǎng)絡(luò)包抓取,采用基于網(wǎng)絡(luò)包五元組的負(fù)載均衡策略對(duì)網(wǎng)絡(luò)包進(jìn)行分組分發(fā),并利用批處理、無鎖隊(duì)列、多線程等技術(shù)將多個(gè)網(wǎng)絡(luò)包封裝為單條大消息并行發(fā)送,改進(jìn)與優(yōu)化網(wǎng)絡(luò)包轉(zhuǎn)發(fā)性能;然后,采用Kafka消息系統(tǒng)作為中間件接收并緩存網(wǎng)絡(luò)包,從而實(shí)現(xiàn)網(wǎng)絡(luò)包的分布式導(dǎo)入;接著,基于Storm搭建實(shí)時(shí)流處理平臺(tái),開發(fā)并部署分布式流重組應(yīng)用,實(shí)現(xiàn)從Kafka中讀取網(wǎng)絡(luò)包,解析并抽取五元組、包大小、時(shí)間戳等信息后重組成網(wǎng)絡(luò)流;最后,增加Hive流數(shù)據(jù)導(dǎo)入模塊,將導(dǎo)出的網(wǎng)絡(luò)流數(shù)據(jù)以Parquet格式實(shí)時(shí)存入HDFS,利用Hive Metastore存儲(chǔ)并管理元數(shù)據(jù),同時(shí)采用基于時(shí)間的動(dòng)態(tài)分區(qū)機(jī)制以減少按時(shí)間檢索時(shí)不必要的磁盤IO。實(shí)驗(yàn)結(jié)果表明:網(wǎng)絡(luò)流量采集模塊可實(shí)現(xiàn)萬兆流量的準(zhǔn)確采集與轉(zhuǎn)發(fā),即使在萬兆流量均為最小包(60字節(jié))的情況下,仍可保證僅有0.03%的丟包率;網(wǎng)絡(luò)流量導(dǎo)入模塊吞吐率與磁盤寫入性能相關(guān),在使用7塊硬盤緩存數(shù)據(jù)時(shí)吞吐率可達(dá)775 MB/s;分布式流重組模塊具有良好的通用性及擴(kuò)展性,通過簡(jiǎn)單配置即可達(dá)到1.26×10~7包/s的吞吐率。目前,NTCI-Flow已用于采集與處理某機(jī)構(gòu)的出口流量,該機(jī)構(gòu)平均流量約3.5 Gbps,峰值帶寬為6 Gbps,每秒包數(shù)最高可達(dá)百萬級(jí)。在該實(shí)際應(yīng)用中,NTCI-Flow運(yùn)行情況良好,由其得到的流量數(shù)據(jù)比Net Stream更準(zhǔn)確。
[Abstract]:In view of the problems existing in the current flow export technology based on software / hardware, such as data distortion and inextensibility, this paper presents an accurate, universal and extensible high-speed network traffic processing framework NTCI-Flow.. First of all, the high performance network packet grab is realized based on PF_RING DNA, and the network packet is distributed by the load balancing strategy based on the five-tuple network packet, and the batch processing is used and the unlocked queue is used. Multithreading encapsulates multiple network packets into a single large message, improves and optimizes the network packet forwarding performance, and then uses Kafka message system as middleware to receive and cache network packets, so as to realize the distributed import of network packets. Then, a real-time stream processing platform based on Storm is built to develop and deploy distributed stream recombination application, which can read network packets from Kafka, parse and extract five-tuple, packet size, timestamp and reorganize into network flow. Adding Hive stream data import module, storing the exported network stream data into HDFS, in Parquet format and using Hive Metastore to store and manage metadata, and adopting the dynamic partitioning mechanism based on time to reduce the unnecessary disk IO. when retrieving by time The experimental results show that the network traffic acquisition module can accurately collect and transmit the ten thousand megabytes, even if the ten thousand megabytes are the smallest packet (60 bytes), only 0.03% of the packet loss rate can be guaranteed. The throughput of the network traffic import module is related to the disk write performance. The throughput of 775 MB/s; distributed stream recombination module has good generality and expansibility when using 7 hard disks to cache data. The throughput of 1.26 脳 10 ~ 7 packets / s can be achieved by simple configuration. At present, NTCI-Flow has been used to collect and process the outlet flow of a certain organization. The average flow of the mechanism is about 3.5 Gbps, with a peak bandwidth of 6 Gbps, / s and the maximum number of packets per second can reach 1 million. In this practical application, NTCI-Flow is running well and the flow data obtained from it are more accurate than Net Stream.
【作者單位】： 四川大學(xué)計(jì)算機(jī)學(xué)院;
【基金】：國(guó)家自然科學(xué)基金資助項(xiàng)目(61272447)
【分類號(hào)】：TP393.08

【相似文獻(xiàn)】

相關(guān)期刊論文 前5條

1 唐磊,金連甫;大型網(wǎng)絡(luò)自動(dòng)信息機(jī)設(shè)計(jì)與實(shí)現(xiàn)[J];計(jì)算機(jī)工程與設(shè)計(jì);2004年08期

2 謝文亮;唐屹;王大星;;基于FPGA技術(shù)的網(wǎng)絡(luò)包頭分類的研究[J];廣州大學(xué)學(xué)報(bào)(自然科學(xué)版);2007年03期

3 BEAN;;我的IRC簡(jiǎn)歷[J];軟件世界;1998年10期

4 張文波,趙海,王小英,關(guān)沫;基于ARMLinux的EWS過載性能研究[J];通信學(xué)報(bào);2005年08期

5 ;[J];;年期

相關(guān)碩士學(xué)位論文 前3條

1 朱新宇;基于鄰居的分布式網(wǎng)絡(luò)異常節(jié)點(diǎn)檢測(cè)定位系統(tǒng)[D];上海交通大學(xué);2015年

2 劉琦;網(wǎng)絡(luò)包接收擁塞問題的研究與解決[D];東北大學(xué);2005年

3 陳金牛;嵌入式IPv6防火墻設(shè)計(jì)與實(shí)現(xiàn)[D];廈門大學(xué);2007年

，

本文編號(hào)：2234683