基于預(yù)測(cè)的Hadoop任務(wù)調(diào)度算法優(yōu)化
發(fā)布時(shí)間：2019-06-19 18:06
【摘要】：在出現(xiàn)落后任務(wù)時(shí),現(xiàn)有的Hadoop推測(cè)式執(zhí)行任務(wù)調(diào)度器會(huì)在空閑節(jié)點(diǎn)上為該任務(wù)進(jìn)行備份執(zhí)行,但并未周全地考慮該空閑節(jié)點(diǎn)的當(dāng)前性能,可能造成這次備份任務(wù)仍然失敗或執(zhí)行速度非常慢,導(dǎo)致執(zhí)行備份任務(wù)的失敗率較高,不僅占用較多系統(tǒng)資源并且延遲了系統(tǒng)響應(yīng)時(shí)間。因此,研究現(xiàn)有的Hadoop任務(wù)調(diào)度算法,針對(duì)備份任務(wù)的調(diào)度執(zhí)行提出改進(jìn)方案,對(duì)提高系統(tǒng)性能有非常重要的意義。提出了基于預(yù)測(cè)的Hadoop任務(wù)調(diào)度優(yōu)化算法——CPL(Computation Prediction of Late)調(diào)度算法,主要包含兩個(gè)優(yōu)化點(diǎn):首先,在系統(tǒng)中維護(hù)兩個(gè)預(yù)測(cè)隊(duì)列,分別為CPU空閑型節(jié)點(diǎn)隊(duì)列和I/O空閑型節(jié)點(diǎn)隊(duì)列。隊(duì)列內(nèi)部按執(zhí)行任務(wù)失敗率升序排序,在匹配任務(wù)和節(jié)點(diǎn)類型的基礎(chǔ)上,預(yù)測(cè)即將空閑且失敗率低的節(jié)點(diǎn)執(zhí)行備份任務(wù),降低了執(zhí)行備份任務(wù)的失敗率;其次,利用Map任務(wù)占用CPU時(shí)間片段的總和對(duì)現(xiàn)有的任務(wù)分類算法進(jìn)行了修正,提出了更加準(zhǔn)確的任務(wù)類型劃分方法。通過Cloud Sim云計(jì)算仿真平臺(tái)進(jìn)行仿真實(shí)驗(yàn),對(duì)CPL調(diào)度算法的性能進(jìn)行了驗(yàn)證。結(jié)果表明:CPL調(diào)度算法的作業(yè)響應(yīng)時(shí)間相比于FIFO調(diào)度算法和LATE調(diào)度算法分別降低了20%和14%;CPL調(diào)度算法比LATE調(diào)度算法執(zhí)行備份任務(wù)的失敗率平均降低了16%。
[Abstract]:when a backward task is present, the existing Hadoop speculative execution task scheduler performs backup execution for the task on the idle node, but does not fully consider the current performance of the idle node, and may cause the backup task to still fail or the execution speed is very slow, Resulting in a high failure rate to perform a backup task, not only more system resources but also system response times. Therefore, the existing Hadoop task scheduling algorithm is studied, and the improvement scheme is put forward for the scheduling of the backup task, which is of great significance to the improvement of the system performance. In this paper, a prediction-based scheduling algorithm for Hadoop task scheduling is proposed, which mainly includes two optimization points: first, the two prediction queues are maintained in the system, and the idle-type node queue and the I/ O idle-type node queue are respectively reserved for the CPU. according to the ascending order of the execution task failure rate in the queue, on the basis of the matching task and the node type, the node which is to be idle and the failure rate is low is predicted to perform the backup task, the failure rate of executing the backup task is reduced, and secondly, By using the sum of the CPU time segments occupied by the Map task, the existing task classification algorithm is modified, and a more accurate method of the task type is proposed. The performance of the CPL scheduling algorithm is verified by the simulation experiment of the Cloud Sim cloud computing simulation platform. The results show that the operation response time of the CPL scheduling algorithm is reduced by 20% and 14%, respectively, compared with the FIFO scheduling algorithm and the LATE scheduling algorithm, and the failure rate of the CPL scheduling algorithm to perform the backup task by the LATE scheduling algorithm is reduced by 16%.
【學(xué)位授予單位】：華中科技大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2016
【分類號(hào)】：TP301.6
本文編號(hào)：2502544
[Abstract]:when a backward task is present, the existing Hadoop speculative execution task scheduler performs backup execution for the task on the idle node, but does not fully consider the current performance of the idle node, and may cause the backup task to still fail or the execution speed is very slow, Resulting in a high failure rate to perform a backup task, not only more system resources but also system response times. Therefore, the existing Hadoop task scheduling algorithm is studied, and the improvement scheme is put forward for the scheduling of the backup task, which is of great significance to the improvement of the system performance. In this paper, a prediction-based scheduling algorithm for Hadoop task scheduling is proposed, which mainly includes two optimization points: first, the two prediction queues are maintained in the system, and the idle-type node queue and the I/ O idle-type node queue are respectively reserved for the CPU. according to the ascending order of the execution task failure rate in the queue, on the basis of the matching task and the node type, the node which is to be idle and the failure rate is low is predicted to perform the backup task, the failure rate of executing the backup task is reduced, and secondly, By using the sum of the CPU time segments occupied by the Map task, the existing task classification algorithm is modified, and a more accurate method of the task type is proposed. The performance of the CPL scheduling algorithm is verified by the simulation experiment of the Cloud Sim cloud computing simulation platform. The results show that the operation response time of the CPL scheduling algorithm is reduced by 20% and 14%, respectively, compared with the FIFO scheduling algorithm and the LATE scheduling algorithm, and the failure rate of the CPL scheduling algorithm to perform the backup task by the LATE scheduling algorithm is reduced by 16%.
【學(xué)位授予單位】：華中科技大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2016
【分類號(hào)】：TP301.6
【參考文獻(xiàn)】
相關(guān)期刊論文 前1條
1 左利云;曹志波;;云計(jì)算中調(diào)度問題研究綜述[J];計(jì)算機(jī)應(yīng)用研究;2012年11期
，本文編號(hào)：2502544
本文鏈接：http://sikaile.net/kejilunwen/ruanjiangongchenglunwen/2502544.html
最近更新
教材專著
