【摘要】：伴隨著互聯(lián)網(wǎng)技術(shù)的飛速發(fā)展,網(wǎng)絡(luò)數(shù)據(jù)規(guī)模急劇膨脹,典型的有社交網(wǎng)絡(luò),比如Twitter每兩天半就能產(chǎn)生十億條推文,每天處理近2TB數(shù)據(jù),電子商務(wù)領(lǐng)域,比如支付寶單日最高成功支付1.88億筆,搜索引擎的代表Google每天處理的數(shù)據(jù)量達到20PB,吉爾德定律也預示著網(wǎng)絡(luò)流量將持續(xù)膨脹。不光網(wǎng)絡(luò)流的數(shù)據(jù)量急劇膨脹,網(wǎng)絡(luò)流量的形式也日趨復雜,隨著CDN技術(shù),多源下載技術(shù)等發(fā)展與普及,網(wǎng)絡(luò)流的形式不再是以往的簡單連接形式,給網(wǎng)絡(luò)流過濾帶來了極大的挑戰(zhàn),目前,CDN廠商Akamai產(chǎn)生的流量占全球流量的40%, YouTube大量使用多源下載技術(shù),其流量占北美流量的30%。 對海量數(shù)據(jù)流進行過濾的DPI系統(tǒng)目前普遍采用分布式多級并行處理的方式對網(wǎng)絡(luò)流進行分析與檢測,分布式集群需要一個高效的負載均衡系統(tǒng)對流量進行分發(fā),本文緊密圍繞DPI系統(tǒng)前端的負載均衡系統(tǒng)的web數(shù)據(jù)整合,數(shù)據(jù)分流,流量調(diào)度等進行相關(guān)技術(shù)研究,開展了一系列的關(guān)鍵技術(shù)研究與系統(tǒng)實現(xiàn)工作。由于CDN技術(shù)以及多源下載技術(shù)的大量使用,導致網(wǎng)絡(luò)流量中的會話可能是有多個連接組成,而往往DPI系統(tǒng)需要將整個會話分流至一臺后端機才能進行完整的分析,目前已有的負載均衡技術(shù)不具備將該類會話保持完整的功能,本文中的web數(shù)據(jù)整合功能為了解決該問題,將同一域名下的服務(wù)端IP聚類成IP簇,以IP簇為分流單元進行分流,從而將同一內(nèi)容提供商的流量匯聚至同一后端機進行分析,一方面解決了數(shù)據(jù)會話完整性的問題,另一方面同一內(nèi)容提供商的內(nèi)容會被眾多用戶訪問,如果每個用戶的訪問的同一內(nèi)容的流量都被后端機分析一次,將造成大量的重復性計算,導致寶貴計算資源的浪費,通過web數(shù)據(jù)的整合,負載均衡系統(tǒng)可以對冗余數(shù)據(jù)進行去重,減少DPI系統(tǒng)的大量重復性計算,從而節(jié)約計算資源,提高DPI系統(tǒng)的吞吐量。數(shù)據(jù)分流和流量調(diào)度是在web數(shù)據(jù)整合的基礎(chǔ)上,以IP簇為流量調(diào)度單元依據(jù)DPI系統(tǒng)的反饋進行負載均衡,伴隨著后端機負載的變化,IP簇實時的進行分裂與合并,從而均衡各個后端機的負載。
[Abstract]:With the rapid development of Internet technology, the scale of network data has expanded dramatically, such as social networks, such as Twitter can generate billions of tweets every two and a half days, deal with nearly 2TB data every day, e-commerce, For example, Alipay pays a maximum of 188 million payments a day, Google, the search engine's representative, handles 20 PBs a day, and Gillard's law indicates that network traffic will continue to swell. Not only the amount of data of network flow expands rapidly, but the form of network flow is becoming more and more complicated. With the development and popularization of CDN technology and multi-source download technology, the form of network flow is no longer the simple connection form in the past. It brings great challenges to network flow filtering. At present, Akamai manufacturers account for 40 percent of global traffic. YouTube uses multi-source download technology, and its traffic accounts for 30 percent of North American traffic. At present, DPI system which filters mass data streams generally uses distributed multilevel parallel processing to analyze and detect network flows. Distributed clusters need an efficient load balancing system to distribute traffic. This paper focuses on the research of web data integration, data flow, traffic scheduling and a series of key technology research and system implementation of load balancing system in front of DPI system. Due to the extensive use of CDN technology and multi-source download technology, the sessions in network traffic may be composed of multiple connections, and often the DPI system needs to split the whole session to a single back-end machine to complete the analysis. The existing load balancing technology does not have the function of keeping the session intact. In order to solve this problem, the web data integration function in this paper clustered the server IP under the same domain name into a IP cluster and split the IP cluster as the shunt unit. Thus, the traffic of the same content provider is aggregated to the same back-end machine for analysis. On the one hand, the problem of data session integrity is solved; on the other hand, the content of the same content provider will be accessed by many users. If the traffic of the same content visited by each user is analyzed once by the back-end machine, it will result in a large number of repetitive calculations, resulting in a waste of valuable computing resources, and the integration of web data. The load balancing system can remove redundant data and reduce a large number of repetitive calculations in DPI system, thus saving computing resources and improving the throughput of DPI system. Data streaming and traffic scheduling is based on the integration of web data, and the IP cluster is used as the traffic scheduling unit to balance the load according to the feedback of the DPI system. With the change of the backend machine load, the IP cluster is split and merged in real time. Thus balancing the load of each backend machine.
【學位授予單位】：北京郵電大學
【學位級別】：碩士
【學位授予年份】：2014
【分類號】：TP393.06

【參考文獻】

相關(guān)期刊論文 前10條

1 蔣江,張民選,廖湘科;基于多種資源的負載平衡算法的研究[J];電子學報;2002年08期

2 楊際祥;譚國真;王榮生;;并行與分布式計算動態(tài)負載均衡策略綜述[J];電子學報;2010年05期

3 陳訓遜,方濱興,李蕾;高速網(wǎng)絡(luò)環(huán)境下入侵檢測系統(tǒng)結(jié)構(gòu)研究[J];計算機研究與發(fā)展;2004年09期

4 班志杰;古志民;金瑜;;Web預取技術(shù)綜述[J];計算機研究與發(fā)展;2009年02期

5 陳志剛,許偉,曾志文;一種基于預測的動態(tài)負載均衡模型及算法研究[J];計算機工程;2004年23期

6 李清玉,韋衛(wèi),侯紫峰;防火墻集群動態(tài)自適應(yīng)負載均衡算法設(shè)計與實現(xiàn)[J];計算機工程;2005年01期

7 孫欽東,張德運,高鵬,劉剛;基于動態(tài)負載均衡的分層式高速網(wǎng)絡(luò)入侵檢測模型[J];計算機工程;2005年12期

8 王成;劉金剛;;一種改進的字符串匹配算法[J];計算機工程;2006年02期

9 袁志堅;陳穎文;繆嘉嘉;賈焰;楊樹強;;典型Bloom過濾器的研究及其數(shù)據(jù)流應(yīng)用[J];計算機工程;2009年07期

10 嵇智輝;倪宏;劉磊;;動態(tài)消息隊列負載均衡策略的研究與應(yīng)用[J];計算機工程;2009年08期

，

本文編號：2200753