本文選題：數(shù)據(jù)流　切入點(diǎn)：流式處理　出處：《計(jì)算機(jī)工程與科學(xué)》2014年10期 　論文類型：期刊論文

【摘要】：流式數(shù)據(jù)處理中,數(shù)據(jù)傾斜等原因易導(dǎo)致計(jì)算節(jié)點(diǎn)的負(fù)載不均衡,降低系統(tǒng)處理能力。傳統(tǒng)的負(fù)載均衡方法,比如算子分配、算子遷移和負(fù)載脫落等技術(shù)因?yàn)橄鄬?duì)較高的性能代價(jià),在流式處理系統(tǒng)中沒(méi)有得到廣泛的應(yīng)用。針對(duì)流式處理系統(tǒng)的特點(diǎn),提出一種新的負(fù)載均衡方法。在該方法中,計(jì)算單元的數(shù)據(jù)被劃分為若干分區(qū),并且數(shù)據(jù)分區(qū)可以在計(jì)算單元中動(dòng)態(tài)分配和遷移,在較少干擾系統(tǒng)運(yùn)行的情況下,通過(guò)動(dòng)態(tài)調(diào)整各計(jì)算單元的分區(qū),平衡各個(gè)計(jì)算單元的輸入流和利用率,以此達(dá)到負(fù)載平衡的目的。在此基礎(chǔ)上,設(shè)計(jì)并實(shí)現(xiàn)了流式處理系統(tǒng)的負(fù)載均衡算法和數(shù)據(jù)在線遷移技術(shù)。實(shí)驗(yàn)結(jié)果表明,該方法能夠顯著減少數(shù)據(jù)處理的平均延遲,提高系統(tǒng)吞吐量。
[Abstract]:In flow data processing, data skew and other reasons can easily lead to the calculation node load imbalance, reduce the system processing ability. Traditional load balancing methods, such as operator allocation, The techniques of operator migration and load shedding have not been widely used in the flow processing system because of the relatively high performance cost. A new load balancing method is proposed according to the characteristics of the flow processing system. The data of the computing unit is divided into several partitions, and the data partition can be dynamically allocated and migrated in the computing unit, and the partition of each computing unit can be dynamically adjusted with less interference with the operation of the system. In order to achieve the purpose of load balancing, the load balancing algorithm and data online migration technology of flow processing system are designed and implemented. The experimental results show that, This method can significantly reduce the average delay of data processing and improve system throughput.
【作者單位】： 北京航空航天大學(xué)計(jì)算機(jī)學(xué)院;泰山學(xué)院物理與電子工程學(xué)院;
【基金】：國(guó)家973計(jì)劃資助項(xiàng)目(2014CB340300)
【分類號(hào)】：TP338.8

【共引文獻(xiàn)】

相關(guān)期刊論文 前10條

1 張浩軍;朱艷琴;紀(jì)其進(jìn);;面向異構(gòu)網(wǎng)絡(luò)的動(dòng)態(tài)負(fù)載均衡算法及其收斂性分析[J];電子與信息學(xué)報(bào);2013年09期

2 劉紀(jì)平;張福浩;王亮;董春;王勇;徐勝華;錢新林;;面向大數(shù)據(jù)的空間信息決策支持服務(wù)研究與展望[J];測(cè)繪科學(xué);2014年05期

3 孫剛;周華平;孫克雷;;基于改進(jìn)的隨機(jī)決策樹的煤礦安全評(píng)價(jià)方法[J];阜陽(yáng)師范學(xué)院學(xué)報(bào)(自然科學(xué)版);2014年02期

4 黃慶榮;;基于大數(shù)據(jù)實(shí)時(shí)洞察客戶感知[J];福建電腦;2014年03期

5 劉小利;徐攀登;朱國(guó)賓;李雪;;結(jié)合MapReduce和HBase的遙感圖像并行分布式查詢[J];地理與地理信息科學(xué);2014年05期

6 楊浩;滕飛;李天瑞;李f，

本文編號(hào)：1625641