【摘要】：當(dāng)下,隨著“云計(jì)算(Cloud Computing)"平臺的火熱發(fā)展,越來越多的高校、研究所、IT公司以及互聯(lián)網(wǎng)企業(yè)開始深入研究并開展云平臺的項(xiàng)目,以求能更好地面對“大數(shù)據(jù)(Big Data)"時(shí)代的來臨。而在這其中,Apache Hadoop作為完全開源的云平臺,受到了大多數(shù)企業(yè),工程師以及專家學(xué)者的青睞,紛紛參與到了Hadoop云計(jì)算平臺的研究和開發(fā)中。 而隨著“云計(jì)算”的火熱發(fā)展,“云服務(wù)”供應(yīng)商正在面臨著越來越巨大,越來越復(fù)雜的數(shù)據(jù)處理。各種PB級別的結(jié)構(gòu)化和非結(jié)構(gòu)化數(shù)據(jù)讓現(xiàn)有的Hadoop平臺處理起來非常地吃力。此時(shí),原生Hadoop在某些特殊作業(yè)的背景下已經(jīng)難以有效地應(yīng)對用戶所提交的各種復(fù)雜任務(wù)了。 本文正是針對目前MapReduce框架下Hadoop現(xiàn)有調(diào)度器在處理大內(nèi)存需求作業(yè)時(shí)出現(xiàn)的任務(wù)等待時(shí)間過長,作業(yè)完成時(shí)間過高等問題,研究了不同調(diào)度器的調(diào)度策略,提出并設(shè)計(jì)了基于模擬退火算法的隊(duì)列級別調(diào)度策略。通過采用隊(duì)列資源利用率作為退火概率,將作業(yè)期望完成時(shí)間、資源量限制等作為設(shè)計(jì)參數(shù),利用模擬退火算法的高效率、低初始條件約束等特點(diǎn),優(yōu)化計(jì)算能力調(diào)度器的調(diào)度效果。本文所做工作如下： 首先,針對目前的Hadoop平臺,分析,研究了Hadoop的設(shè)計(jì)理念,運(yùn)行機(jī)制,掌握了MapReduce的處理框架,并對Hadoop現(xiàn)有調(diào)度器進(jìn)行了深入的學(xué)習(xí),包括Hadoop默認(rèn)的FIFO先進(jìn)先出調(diào)度器,Hadoop中自帶的公平調(diào)度器,計(jì)算能力調(diào)度器,以及在MapReduce事項(xiàng)列表中正式提出且已設(shè)計(jì)出但尚未在Hadoop2.0之前的版本中正式使用的資源感知調(diào)度器和自適應(yīng)調(diào)度器。針對以上五種調(diào)度器,探討了它們的設(shè)計(jì)理念,并對它們的調(diào)度機(jī)理進(jìn)行了研究和分析,指出了目前各種調(diào)度器中所存在的不同問題。 然后,根據(jù)之前的工作中所總結(jié)的在現(xiàn)有各種調(diào)度器中所存在的普遍問題,本文提出并設(shè)計(jì)了一種新型的調(diào)度器,能有效地解決之前調(diào)度器中所存在的對大內(nèi)存需求作業(yè)調(diào)度吃緊的問題。設(shè)計(jì)思路采用改進(jìn)型的模擬退火算法,首先對傳統(tǒng)的模擬退火算法進(jìn)行了分析,之后對如何在調(diào)度器中應(yīng)用給出了改進(jìn)方法,根據(jù)Hadoop平臺下的調(diào)度器原理進(jìn)行了基于模擬退火算法新型調(diào)度策略的設(shè)計(jì)并依據(jù)該策略開發(fā)了新型的Hadoop調(diào)度器。 最后,本文對新型調(diào)度器進(jìn)行了實(shí)際情況測試,包括Hadoop中實(shí)現(xiàn)調(diào)度器的自由切換,針對不同類型作業(yè)的調(diào)度情況測試,在同一種作業(yè)下與計(jì)算能力調(diào)度器的調(diào)度對比測試等等。經(jīng)過實(shí)驗(yàn)驗(yàn)證,本文所設(shè)計(jì)的新型調(diào)度器對大內(nèi)存需求作業(yè)進(jìn)行調(diào)度時(shí)能有效地降低任務(wù)等待情況的發(fā)生,實(shí)現(xiàn)了更低的作業(yè)完成時(shí)間以及更好的資源利用率。基本實(shí)現(xiàn)了hadoop調(diào)度器所需要的功能,同時(shí)也能滿足特殊情況下作業(yè)的合理調(diào)度。
[Abstract]:Nowadays, with the development of cloud computing (Cloud Computing) platform, more and more universities, research institutes, IT companies and Internet enterprises begin to research and develop cloud platform projects in order to better face the "big data (Big Data)" era. As a completely open source cloud platform, Apache Hadoop has been favored by most enterprises, engineers and experts, and has participated in the research and development of Hadoop cloud computing platform. With the development of cloud computing, cloud service providers are facing more and more huge and complex data processing. Various PB-level structured and unstructured data make the existing Hadoop platform very difficult to handle. At this point, native Hadoop in the context of some special jobs has been difficult to effectively deal with the user submitted a variety of complex tasks. In this paper, the scheduling strategies of different schedulers are studied in order to solve the problems such as too long waiting time and too high job completion time when the current Hadoop scheduler processes jobs with large memory requirements under the current MapReduce framework. A queue level scheduling strategy based on simulated annealing algorithm is proposed and designed. By using queue resource utilization as annealing probability, the expected completion time and resource limit are taken as design parameters, and the high efficiency and low initial constraints of simulated annealing algorithm are used. Optimize the scheduling effect of the computing power scheduler. The work of this paper is as follows: firstly, according to the current Hadoop platform, the design concept and running mechanism of Hadoop are studied, the processing framework of MapReduce is mastered, and the existing Hadoop scheduler is deeply studied. Including Hadoop default FIFO first-in first-out scheduler Hadoop comes with a fair scheduler, computing power scheduler, And the resource aware scheduler and adaptive scheduler which are formally put forward in the list of MapReduce items and which have been designed but have not been formally used in the previous version of Hadoop2.0. In view of the above five kinds of schedulers, this paper discusses their design ideas, studies and analyzes their scheduling mechanism, and points out the different problems existing in the various schedulers at present. Then, according to the common problems existing in all kinds of schedulers summarized in previous work, this paper proposes and designs a new kind of scheduler. It can effectively solve the problem of tight job scheduling for large memory requirements in the previous scheduler. The improved simulated annealing algorithm is adopted in the design. Firstly, the traditional simulated annealing algorithm is analyzed, and then the improved method is given for its application in the scheduler. According to the principle of Hadoop scheduler, a new scheduling strategy based on simulated annealing algorithm is designed and a new Hadoop scheduler is developed. Finally, this paper tests the actual situation of the new scheduler, including the implementation of free switching of scheduler in Hadoop, the scheduling test for different types of jobs, the scheduling comparison test between the scheduler and the computing power scheduler under the same kind of job, and so on. Experimental results show that the new scheduler designed in this paper can effectively reduce the occurrence of task waiting and achieve lower job completion time and better resource utilization when scheduling jobs with large memory requirements. The functions of hadoop scheduler are basically realized, and the reasonable scheduling of jobs under special circumstances is also satisfied.
【學(xué)位授予單位】：太原理工大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2014
【分類號】：TP393.05

【參考文獻(xiàn)】

相關(guān)期刊論文 前10條

1 董麗麗;龔光紅;李妮;孫勇;;基于云模型的自適應(yīng)并行模擬退火遺傳算法[J];北京航空航天大學(xué)學(xué)報(bào);2011年09期

2 孫大為;常桂然;李鳳云;王川;王興偉;;一種基于免疫克隆的偏好多維QoS云資源調(diào)度優(yōu)化算法[J];電子學(xué)報(bào);2011年08期

3 鄭世明;高志年;韋偉;苗壯;邵榮明;;基于云模型的網(wǎng)格任務(wù)調(diào)度遺傳算法研究[J];電子科技大學(xué)學(xué)報(bào);2012年06期

4 俞能海;郝卓;徐甲甲;張衛(wèi)明;張馳;;云安全研究進(jìn)展綜述[J];電子學(xué)報(bào);2013年02期

5 李陶深;張希翔;;云計(jì)算下區(qū)分服務(wù)的演化博弈調(diào)度算法[J];北京郵電大學(xué)學(xué)報(bào);2013年01期

6 徐潔;朱健琛;魯珂;;基于雙適應(yīng)度遺傳退火的云任務(wù)調(diào)度算法[J];電子科技大學(xué)學(xué)報(bào);2013年06期

7 林偉偉;齊德昱;;云計(jì)算資源調(diào)度研究綜述[J];計(jì)算機(jī)科學(xué);2012年10期

8 熊聰聰;馮龍;陳麗仙;蘇靜;;云計(jì)算中基于遺傳算法的任務(wù)調(diào)度算法研究[J];華中科技大學(xué)學(xué)報(bào)(自然科學(xué)版);2012年S1期

9 張希翔;李陶深;;云計(jì)算下適應(yīng)用戶任務(wù)動態(tài)變更的調(diào)度算法[J];華中科技大學(xué)學(xué)報(bào)(自然科學(xué)版);2012年S1期

10 李兵;付新s，

本文編號：2147754