發(fā)布時間：2018-01-13 12:29

  本文關(guān)鍵詞：面向海量數(shù)據(jù)的MapReduce本地優(yōu)先作業(yè)調(diào)度策略研究與實(shí)現(xiàn)　出處：《國防科學(xué)技術(shù)大學(xué)》2012年碩士論文　論文類型：學(xué)位論文

  更多相關(guān)文章： 負(fù)載均衡 數(shù)據(jù)本地性 MapReduce 云計(jì)算

【摘要】：近幾十年來，信息網(wǎng)絡(luò)的技術(shù)和規(guī)模都不斷發(fā)展，海量數(shù)據(jù)應(yīng)用不斷增加，由單個企業(yè)搭建的普通計(jì)算集群已難以解決不斷增長的海量數(shù)據(jù)給有效管理和高效計(jì)算帶來的挑戰(zhàn)，因此工業(yè)界提出將計(jì)算推至云端的思想，即云計(jì)算當(dāng)前，云計(jì)算的概念已經(jīng)被企業(yè)和科研機(jī)構(gòu)所廣泛接受，，并且在可靠性可用性等方面取得了很多成果 在這些成果中，MapReduce是海量數(shù)據(jù)分布式計(jì)算中具有重要意義的解決方案之一，它的核心功能已在Hadoop分布式計(jì)算系統(tǒng)中得到實(shí)現(xiàn)Hadoop的開源特性，使得其成為研究MapReduce分布式計(jì)算的重要基礎(chǔ)平臺本文的工作即基于此平臺 MapReduce分布式計(jì)算模型中的作業(yè)調(diào)度問題對系統(tǒng)的性能可靠性等方面具有重要的影響本文針對多作業(yè)情況下現(xiàn)有的作業(yè)調(diào)度算法的數(shù)據(jù)本地性差的問題，提出了一種基于本地優(yōu)先的作業(yè)調(diào)度算法該方法通過新的思路解決數(shù)據(jù)本地性和系統(tǒng)負(fù)載均衡性相沖突的問題，在保證數(shù)據(jù)本地性的同時，通過作業(yè)級別的調(diào)度優(yōu)化系統(tǒng)的負(fù)載均衡性能，降低了計(jì)算過程中的IO開銷，從而增加系統(tǒng)的吞吐率和減少單個作業(yè)的執(zhí)行時間 本文在以HDFS為分布式存儲系統(tǒng)的MapReduce編程模型中設(shè)計(jì)實(shí)現(xiàn)了基于本地優(yōu)先的作業(yè)調(diào)度算法，并且在仿真環(huán)境中進(jìn)行了實(shí)驗(yàn)驗(yàn)證實(shí)驗(yàn)結(jié)果顯示，在完全實(shí)現(xiàn)數(shù)據(jù)本地性的機(jī)制下，系統(tǒng)的吞吐率得到有效提升的同時，單個作業(yè)的平均執(zhí)行時間也大大減少
[Abstract]:In recent decades, information network technology and scale development, the increasing use of massive data, common computing cluster from single enterprise has been difficult to solve massive data growing brings to the effective management and efficient computing challenges, so the industry will push to put forward the idea of cloud computing, cloud computing is the current. The concept of cloud computing has been widely accepted by enterprises and research institutions, and made a lot of achievements in reliability, usability etc.
In these results, MapReduce is one of the solutions is of great significance for massive data in distributed computing, its core function has been calculated in Hadoop distributed implementation of Hadoop open source system, making it become the important work platform in MapReduce distributed computing is based on this platform
This paper has the important effect of MapReduce distributed computing scheduling problem in the model performance of the reliability of the system and other aspects of the existing scheduling algorithms work in case of data locality difference problem, put forward a new idea by scheduling algorithm based on local priority based on the solution of data locality and load system the balance of conflict problems, while ensuring the data locality, through load balancing performance scheduling optimization system operation level, reduces the calculation of the IO overhead, thereby increasing system throughput and reduce the execution time of a single job
The design and implementation of scheduling algorithm based on local priority based on the HDFS MapReduce programming model for distributed storage system, and verified the experimental results shown in the simulation environment, in the full realization mechanism of data locality, and effectively improve the system throughput, the average execution time of single job is greatly reduced

【學(xué)位授予單位】：國防科學(xué)技術(shù)大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2012
【分類號】：TP333;TP311.13

【參考文獻(xiàn)】

相關(guān)期刊論文 前4條

1 李明;胥光輝;戢瑤;;MapReduce編程模型在網(wǎng)絡(luò)I/O密集型程序中的應(yīng)用研究[J];計(jì)算機(jī)應(yīng)用研究;2011年09期

2 杜建成,黃皓,陳道蓄,謝立;基于最佳并行度的任務(wù)依賴圖調(diào)度[J];軟件學(xué)報;1999年10期

3 陳康;鄭緯民;;云計(jì)算:系統(tǒng)實(shí)例與研究現(xiàn)狀[J];軟件學(xué)報;2009年05期

4 林子雨;賴永炫;林琛;謝怡;鄒權(quán);;云數(shù)據(jù)庫研究[J];軟件學(xué)報;2012年05期

相關(guān)博士學(xué)位論文 前2條

1 方雷;基于云計(jì)算的土地資源服務(wù)高效處理平臺關(guān)鍵技術(shù)探索與研究[D];浙江大學(xué);2011年

2 陳海波;云計(jì)算平臺可信性增強(qiáng)技術(shù)的研究[D];復(fù)旦大學(xué);2008年

本文編號：1418846