本文選題：云計(jì)算 + MapReduce　； 參考：《南京郵電大學(xué)》2017年碩士論文

【摘要】：隨著信息技術(shù)的快速發(fā)展和普及應(yīng)用,大規(guī)模的數(shù)據(jù)處理需求日益增加,傳統(tǒng)的并行計(jì)算機(jī)在存儲(chǔ)空間和計(jì)算資源方面難以滿足海量數(shù)據(jù)處理的需求,因此云計(jì)算技術(shù)為解決海量數(shù)據(jù)處理提供了良好的環(huán)境。MapReduce編程模型為云計(jì)算進(jìn)行海量數(shù)據(jù)處理提供了新的思路,它克服了傳統(tǒng)分布式并行程序編寫復(fù)雜的缺點(diǎn),提供了簡(jiǎn)化的編寫方式。但是MapReduce編程模型的性能上仍然存在一些問題,因此本文從任務(wù)調(diào)度的角度對(duì)MapReduce的性能進(jìn)行優(yōu)化。本文研究的主要內(nèi)容是:首先對(duì)云計(jì)算和Hadoop平臺(tái)進(jìn)行簡(jiǎn)單介紹,重點(diǎn)研究MapReduce計(jì)算模型和任務(wù)調(diào)度算法,在分析MapReduce調(diào)度算法不足的基礎(chǔ)上,提出了蟻群模擬退火算法,在該算法中,以減少任務(wù)的完成時(shí)間和保證資源負(fù)載均衡為目標(biāo),根據(jù)蟻群算法構(gòu)造局部最優(yōu)解,將蟻群算法的局部最優(yōu)解作為模擬退火算法的初始解進(jìn)行全局搜索,并根據(jù)Metropolis準(zhǔn)則判斷是否接受當(dāng)前解。針對(duì)MapReduce編程模型中的容錯(cuò)技術(shù)的缺點(diǎn),本文提出引入失效恢復(fù)機(jī)制的可靠性任務(wù)調(diào)度策略,對(duì)云環(huán)境中的資源節(jié)點(diǎn)進(jìn)行可信任度評(píng)估,建立可信任度模型,根據(jù)可信任度模型為任務(wù)分配可靠的節(jié)點(diǎn),避免任務(wù)失敗重新分配執(zhí)行,浪費(fèi)時(shí)間和資源。最后,通過仿真平臺(tái)CloudSim驗(yàn)證了本文提出的任務(wù)調(diào)度算法和調(diào)度模型的有效性和穩(wěn)定性。
[Abstract]:With the rapid development and widespread application of information technology, the demand for large-scale data processing is increasing day by day. Traditional parallel computers are unable to meet the requirements of mass data processing in storage space and computing resources. Therefore, cloud computing technology provides a good environment for mass data processing. MapReduce programming model provides a new way for cloud computing to process mass data. It overcomes the complex shortcomings of traditional distributed parallel programs. A simplified way of writing is provided. However, there are still some problems in the performance of MapReduce programming model, so this paper optimizes the performance of MapReduce from the perspective of task scheduling. The main contents of this paper are as follows: firstly, the cloud computing and Hadoop platforms are briefly introduced, and the MapReduce computing model and task scheduling algorithm are studied. Based on the analysis of the lack of MapReduce scheduling algorithm, an ant colony simulated annealing algorithm is proposed. In order to reduce the completion time of the task and ensure the resource load balance, the local optimal solution of the ant colony algorithm is constructed according to the ant colony algorithm, and the local optimal solution of the ant colony algorithm is considered as the initial solution of the simulated annealing algorithm. The Metropolis criterion is used to determine whether the current solution is acceptable or not. Aiming at the shortcomings of fault-tolerant technology in MapReduce programming model, this paper proposes a reliability task scheduling strategy based on failure recovery mechanism, evaluates the trust degree of resource nodes in cloud environment, and establishes a trust degree model. A reliable node is assigned to the task according to the trust model to avoid the task failure redistribution and waste of time and resources. Finally, the effectiveness and stability of the proposed task scheduling algorithm and scheduling model are verified by the simulation platform CloudSim.
【學(xué)位授予單位】：南京郵電大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2017
【分類號(hào)】：TP3;TP18

【參考文獻(xiàn)】

相關(guān)期刊論文 前10條

1 鄭曉薇;項(xiàng)明;張大為;劉青昆;;基于節(jié)點(diǎn)能力的Hadoop集群任務(wù)自適應(yīng)調(diào)度方法[J];計(jì)算機(jī)研究與發(fā)展;2014年03期

2 傅杰;都志輝;;一種周期性MapReduce作業(yè)的負(fù)載均衡策略[J];計(jì)算機(jī)科學(xué);2013年03期

3 許丞;劉洪;譚良;;Hadoop云平臺(tái)的一種新的任務(wù)調(diào)度和監(jiān)控機(jī)制[J];計(jì)算機(jī)科學(xué);2013年01期

4 劉瑋;王麗宏;;云計(jì)算應(yīng)用及其安全問題研究[J];計(jì)算機(jī)研究與發(fā)展;2012年S2期

5 李奇原;劉杰;葉丹;許舒人;;FlowS:一種MapReduce數(shù)據(jù)流公平調(diào)度方法[J];計(jì)算機(jī)科學(xué);2012年09期

6 余正祥;;基于學(xué)習(xí)方式對(duì)Hadoop作業(yè)調(diào)度的改進(jìn)研究[J];計(jì)算機(jī)科學(xué);2012年S1期

7 崔杰;李陶深;蘭紅星;;基于Hadoop的海量數(shù)據(jù)存儲(chǔ)平臺(tái)設(shè)計(jì)與開發(fā)[J];計(jì)算機(jī)研究與發(fā)展;2012年S1期

8 李建江;崔健;王聃;嚴(yán)林;黃義雙;;MapReduce并行編程模型研究綜述[J];電子學(xué)報(bào);2011年11期

9 李麗英;唐卓;李仁發(fā);;基于LATE的Hadoop數(shù)據(jù)局部性改進(jìn)調(diào)度算法[J];計(jì)算機(jī)科學(xué);2011年11期

10 顧宇;周良;丁秋林;;基于優(yōu)先級(jí)的Three-Queue調(diào)度算法研究[J];計(jì)算機(jī)科學(xué);2011年S1期

相關(guān)博士學(xué)位論文 前1條

1 王玉林;多節(jié)點(diǎn)容錯(cuò)存儲(chǔ)系統(tǒng)的數(shù)據(jù)與緩存組織研究[D];電子科技大學(xué);2010年

相關(guān)碩士學(xué)位論文 前3條

1 陸路;云環(huán)境下作業(yè)調(diào)度算法研究[D];南京理工大學(xué);2013年

2 趙春燕;云環(huán)境下作業(yè)調(diào)度算法研究與實(shí)現(xiàn)[D];北京交通大學(xué);2009年

3 汪采萍;蟻群算法的應(yīng)用研究[D];合肥工業(yè)大學(xué);2007年

，

本文編號(hào)：2100808