本文關鍵詞：面向異構多核系統(tǒng)的并行計算模型和調(diào)度算法研究　出處：《湖南大學》2012年碩士論文　論文類型：學位論文

  更多相關文章： 異構多核系統(tǒng) 并行編程模型 MapReduce 推測執(zhí)行 調(diào)度算法

【摘要】：隨著異構多核并行編程的難度不斷增大，人們迫切希望并行編程模型可以處理并能生成超大規(guī)模(TB級)數(shù)據(jù)集，以減少并行編程難度，提高異構多核系統(tǒng)開發(fā)速度。 MapReduce是近些年新興的并行編程模型，該模型主要用于實現(xiàn)并行計算中子任務劃分、資源的調(diào)度、計算結構歸約等，其為異構并行系統(tǒng)的大規(guī)模數(shù)據(jù)處理提供一個簡單、有效的解決方案。然而傳統(tǒng)的MapReduce調(diào)度算法存在任務響應時間過長，系統(tǒng)吞吐量大幅度下降的情況，從而影響整個系統(tǒng)的效率的提高。本文在對MapReduce并行編程模型深入研究的基礎上，提出了一種適應于Hadoop平臺的異構多核的MapReduce調(diào)度改進算法。主要工作如下： (1)針對MapReduce模型的調(diào)度問題，研究了影響MapReduce調(diào)度性能的三個主要因素：本地化、同步開銷及公平性約束，并對處理這三個因素的調(diào)度方法進行分析。對MapReduce模型中同步開銷問題的兩種解決方法：異步處理和推測執(zhí)行進行了探究。對于公平性約束，討論了Hadoop的本地提升和延遲調(diào)度，以及Dryad的Quincy調(diào)度器。 (2)結合異構多核環(huán)境的特性，針對基于典型MapReduce調(diào)度算法——LATE算法的不足，提出了一種MapReduce異構多核調(diào)度的改進算法，該算法通過在系統(tǒng)上添加使系統(tǒng)獲得自動學習的能力——機器學習中的監(jiān)管學習，隨機提取部分工作任務作為測試任務，以獲得處理節(jié)點的處理信息，進而得到任務處理的各個階段的實際時間比，并調(diào)整程序的運行方式，從而啟動備份任務，以提高任務響應時間。 為了驗證本文算法的有效性，本文在Hadoop平臺基礎上，對本文算法進行了實驗，實驗結果表明本文算法在任務響應時間上，，優(yōu)于LATE算法和Hadoop平臺原有調(diào)度算法，有利于整個系統(tǒng)處理效率的提高，對異構多核并行計算具有一定的推動意義。
[Abstract]:With the increasing difficulty of heterogeneous multi-core parallel programming, people urgently hope that the parallel programming model can process and generate large scale / terabyte (TB) data sets, so as to reduce the difficulty of parallel programming. Improve the development speed of heterogeneous multi-core system. MapReduce is a new parallel programming model in recent years. This model is mainly used to realize the parallel computing neutron task partition, resource scheduling, computing structure reduction and so on. It provides a simple and effective solution for large-scale data processing in heterogeneous parallel systems. However, the task response time of traditional MapReduce scheduling algorithm is too long. The throughput of the system is greatly reduced, which affects the efficiency of the whole system. This paper deeply studies the parallel programming model of MapReduce. In this paper, an improved MapReduce scheduling algorithm based on heterogeneous multicore for Hadoop platform is proposed. The main work is as follows: 1) aiming at the scheduling problem of MapReduce model, three main factors affecting the scheduling performance of MapReduce are studied: localization, synchronization overhead and fairness constraints. This paper also analyzes the scheduling methods to deal with these three factors, and explores two solutions to the synchronous overhead problem in the MapReduce model: asynchronous processing and speculative execution. The local promotion and delay scheduling of Hadoop and the Quincy scheduler of Dryad are discussed. 2) considering the characteristics of heterogeneous multi-core environment, aiming at the shortcomings of the typical MapReduce scheduling algorithm, path algorithm. In this paper, an improved algorithm for heterogeneous multi-core scheduling of MapReduce is proposed. The algorithm adds the ability of automatic learning to the system, which is the supervised learning in machine learning. A part of the task is randomly extracted as a test task to obtain the processing information of the processing node, and then the actual time ratio of each stage of the task processing is obtained, and the operation mode of the program is adjusted to start the backup task. To increase task response time. In order to verify the effectiveness of this algorithm, this paper based on the Hadoop platform, the experimental results show that the algorithm in the task response time. It is superior to the LATE algorithm and the original scheduling algorithm of Hadoop platform, which is beneficial to the improvement of the processing efficiency of the whole system, and has a certain significance to promote the heterogeneous multi-core parallel computing.
【學位授予單位】：湖南大學
【學位級別】：碩士
【學位授予年份】：2012
【分類號】：TP338.6

【參考文獻】

相關期刊論文 前7條

1 鄭欣杰;朱程榮;熊齊邦;;基于MapReduce的分布式光線跟蹤的設計與實現(xiàn)[J];計算機工程;2007年22期

2 陳全;鄧倩妮;;異構環(huán)境下自適應的Map-Reduce調(diào)度[J];計算機工程與科學;2009年S1期

3 周鋒;李旭偉;;一種改進的MapReduce并行編程模型[J];科協(xié)論壇(下半月);2009年02期

4 陳康;鄭緯民;;云計算:系統(tǒng)實例與研究現(xiàn)狀[J];軟件學報;2009年05期

5 鄭啟龍;王昊;吳曉偉;房明;;HPMR:多核集群上的高性能計算支撐平臺[J];微電子學與計算機;2008年09期

6 王鄂;李銘;;云計算下的海量數(shù)據(jù)挖掘研究[J];現(xiàn)代計算機(專業(yè)版);2009年11期

7 陳國良;苗乾坤;孫廣中;徐云;鄭啟龍;;分層并行計算模型[J];中國科學技術大學學報;2008年07期

相關博士學位論文 前1條

1 張琦;多核系統(tǒng)中的程序性能優(yōu)化研究[D];中國科學技術大學;2010年

相關碩士學位論文 前1條

1 胡利軍;Web集群服務器的負載均衡和性能優(yōu)化[D];北京郵電大學;2010年

本文編號：1420674