本文選題：流式計算 + Storm�。� 參考：《南京郵電大學(xué)》2017年碩士論文

【摘要】：隨著互聯(lián)網(wǎng)業(yè)務(wù)數(shù)據(jù)規(guī)模的急劇增加,人們處理和使用數(shù)據(jù)的模式已經(jīng)發(fā)生了翻天覆地的變化,為了滿足人們對數(shù)據(jù)實時處理能力越來越高的需求,流式計算應(yīng)運(yùn)而生。Storm平臺是一個對實時數(shù)據(jù)進(jìn)行處理的開源平臺,它能夠快速可靠地處理流式數(shù)據(jù),滿足人們越來越迫切的需求。Storm平臺的調(diào)度器是Storm平臺的核心技術(shù)之一,對Storm集群的性能和資源利用有著直接的影響。因此對Storm平臺的調(diào)度器進(jìn)行研究和改進(jìn)對Storm平臺的發(fā)展有著重大的意義。論文主要工作如下:首先,介紹了流式計算和Storm平臺的相關(guān)知識及國內(nèi)外研究現(xiàn)狀,重點研究了分布式開源流式平臺Storm的整體架構(gòu)和作業(yè)處理相關(guān)核心技術(shù)。然后對Storm平臺提供的調(diào)度器(默認(rèn)調(diào)度器、均衡調(diào)度器、隔離調(diào)度器)分別進(jìn)行深度分析;不僅通過一個示例分析提交到Storm平臺的同一作業(yè)在調(diào)用三種不同的調(diào)度器時任務(wù)分配的表現(xiàn),總結(jié)出各個調(diào)度器的特點、適用場景和存在的問題;還著設(shè)置Storm任務(wù)調(diào)度性能的評估指標(biāo),通過實驗對Storm默認(rèn)調(diào)度器的任務(wù)分配存在的問題進(jìn)行分析。接著,針對默認(rèn)調(diào)度器在分配任務(wù)時不僅會忽略了節(jié)點間和進(jìn)程間通信,還忽略了作業(yè)的結(jié)構(gòu)及任務(wù)對資源的實際需求和Storm集群工作節(jié)點的資源狀態(tài)等問題,提出了基于蟻群算法的資源感知任務(wù)調(diào)度算法(RSBA),該算法在調(diào)度的過程中將工作節(jié)點的資源動態(tài)變化表示為螞蟻運(yùn)動所需的信息素,覓食的螞蟻身上帶著自身資源需求標(biāo)簽,任務(wù)調(diào)度過程類似螞蟻覓食過程,對原有Storm默認(rèn)調(diào)度策略進(jìn)行改進(jìn)與優(yōu)化。最后,對基于蟻群算法的資源感知任務(wù)調(diào)度算法(RSBA)進(jìn)行實驗驗證。實驗結(jié)果表明該算法學(xué)習(xí)能力強(qiáng),可找到與當(dāng)前任務(wù)所需資源最匹配的節(jié)點,達(dá)到了合理分配資源的目的。相對于Storm平臺的默認(rèn)調(diào)度算法,RSBA算法不僅可以提高任務(wù)調(diào)度的效率、有效減少作業(yè)平均處理時間、提高Storm集群的吞吐量,而且有利于集群的負(fù)載均衡、可優(yōu)化Storm集群性能。
[Abstract]:With the rapid increase in the scale of Internet business data, the mode of data processing and using has changed dramatically. In order to meet the increasing demand for real-time data processing capacity, Streaming computing is an open source platform for real-time data processing. It can process streaming data quickly and reliably. The scheduler of the platform. Storm is one of the core technologies of Storm platform. It has a direct impact on the performance and resource utilization of Storm clusters. Therefore, it is of great significance to research and improve the scheduler of Storm platform for the development of Storm platform. The main work of this paper is as follows: firstly, the related knowledge of flow computing and Storm platform and the current research situation at home and abroad are introduced, and the whole architecture and core technology of job processing of distributed open source streaming platform Storm are studied emphatically. Then the scheduler (default scheduler, equalization scheduler, isolated scheduler) provided by Storm platform is analyzed in depth. Not only through an example analysis of the same job submitted to the Storm platform in the call of three different scheduler task assignment performance, summed up the characteristics of each scheduler, applicable scenarios and existing problems; The evaluation index of Storm task scheduling performance is also set, and the problems existing in the task allocation of Storm default scheduler are analyzed through experiments. Then, the default scheduler not only ignores the communication between nodes and processes, but also ignores the structure of the job, the actual resource requirement of the task and the resource status of the Storm cluster work node. A resource aware task scheduling algorithm based on ant colony algorithm (ant colony algorithm) is proposed in this paper. In the process of scheduling, the resource dynamic change of the working node is expressed as the pheromone needed by the ant movement, and the foraging ant has its own resource requirement label. The task scheduling process is similar to the ant foraging process. The original Storm default scheduling strategy is improved and optimized. Finally, the resource aware task scheduling algorithm based on ant colony algorithm (RSBA) is tested. The experimental results show that the algorithm has strong learning ability and can find the node that is the best match to the resource needed by the current task, and achieve the purpose of allocating resources reasonably. Compared with the default scheduling algorithm of Storm platform, it can not only improve the efficiency of task scheduling, reduce the average processing time of jobs, improve the throughput of Storm cluster, but also help to balance the load of Storm cluster and optimize the performance of Storm cluster.
【學(xué)位授予單位】：南京郵電大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2017
【分類號】：TP18;TP301.6

【參考文獻(xiàn)】

相關(guān)期刊論文 前9條

1 熊安萍;王賢穩(wěn);鄒洋;;基于Storm拓?fù)浣Y(jié)構(gòu)熱邊的調(diào)度算法[J];計算機(jī)工程;2017年01期

2 王潤華;毋建軍;侯佳路;;分布式實時計算引擎——Storm研究[J];中國科技信息;2015年06期

3 覃雄派;王會舉;杜小勇;王珊;;大數(shù)據(jù)分析——RDBMS與MapReduce的競爭與共生[J];軟件學(xué)報;2012年01期

4 賀曉麗;;一種用于任務(wù)調(diào)度的廣義遺傳算法[J];計算機(jī)工程;2010年17期

5 許昌;常會友;徐俊;衣楊;;一種新的融合分布估計的蟻群優(yōu)化算法[J];計算機(jī)科學(xué);2010年02期

6 黃亞平;熊婧;;基于改進(jìn)蟻群算法作業(yè)車間調(diào)度問題仿真研究[J];計算機(jī)仿真;2009年08期

7 鄭勇明;彭鳳梅;陳越;;分布式數(shù)據(jù)庫查詢優(yōu)化處理——基于關(guān)系代數(shù)等價變換的查詢優(yōu)化處理[J];電腦知識與技術(shù)(學(xué)術(shù)交流);2007年04期

8 高尚;解旅行商問題的混沌蟻群算法[J];系統(tǒng)工程理論與實踐;2005年09期

9 段國林,查建中,徐安平,張滿囤;啟發(fā)式算法及其在工程中的應(yīng)用[J];機(jī)械設(shè)計;2000年06期

，

本文編號：1908336