【摘要】：隨著云計(jì)算技術(shù)的發(fā)展,各種大小不一的數(shù)據(jù)中心紛紛出現(xiàn),而這些數(shù)據(jù)中心往往存在各種虛擬機(jī)管理平臺(如Eucalyptus, OpenNebula和OpenStack等),應(yīng)用場景需求也完全不同,各種管理平臺要求不同的運(yùn)維、開發(fā)技術(shù)和經(jīng)驗(yàn),不同管理平臺問的服務(wù)器資源不能動態(tài)共享,影響了彈性服務(wù)的性能。同時(shí)由于平臺中的不同的機(jī)器配置進(jìn)而也將影響其上層運(yùn)行的云計(jì)算應(yīng)用。Hadoop作為已廣泛應(yīng)用于數(shù)據(jù)密集型計(jì)算的云計(jì)算應(yīng)用之一,其中的MapReduce框架可配置參數(shù)的正確配置對計(jì)算的性能有著不可忽視的影響。然而,當(dāng)遇到異構(gòu)的Hadoop集群時(shí),用戶一般只能使用默認(rèn)配置或者依照經(jīng)驗(yàn)進(jìn)行手工配置,由于參數(shù)調(diào)優(yōu)時(shí)可選擇的空間很大,這樣常常會導(dǎo)致錯(cuò)誤地配置致使計(jì)算性能下降。 針對多種多樣的虛擬機(jī)平臺的問題,本文設(shè)計(jì)并實(shí)現(xiàn)了一個(gè)異構(gòu)虛擬機(jī)管理平臺。在不需要改變既有的虛擬機(jī)管理平臺結(jié)構(gòu)的基礎(chǔ)上,實(shí)現(xiàn)對現(xiàn)有主流的虛擬機(jī)管理平臺的統(tǒng)一管理和控制、虛擬資源的均衡分配；同時(shí)還提供可擴(kuò)展的適配層接口和驅(qū)動部件,支持其它異構(gòu)的虛擬機(jī)供應(yīng)和管理平臺。 針對異構(gòu)虛擬機(jī)平臺上的Hadoop應(yīng)用的問題,本文提出了一種基于增強(qiáng)學(xué)習(xí)的MapReduce在線參數(shù)自動配置方法。該方法利用離線學(xué)習(xí)粗粒度地創(chuàng)建初始化策略,在線學(xué)習(xí)根據(jù)策略細(xì)粒度地配置參數(shù),并通過試錯(cuò)法迭代地更新Q值表使得配置結(jié)果接近最優(yōu)。實(shí)驗(yàn)結(jié)果表明,該配置方法可以有效地提高Hadoop的性能,并且能快速迭代實(shí)現(xiàn)收斂,使運(yùn)行MapReduce任務(wù)的機(jī)器資源得到充分使用,縮短任務(wù)的運(yùn)行時(shí)間。
[Abstract]:With the development of cloud computing technology, a variety of data centers of different sizes have emerged, and these data centers often have a variety of virtual machine management platforms (such as Eucalyptus, OpenNebula and OpenStack), and the requirements of application scenarios are completely different. Different management platforms require different operation and maintenance, development technology and experience. The server resources of different management platforms can not be dynamically shared, which affects the performance of flexible services. At the same time, because of the different machine configuration in the platform, the cloud computing application. Hadoop, which affects the upper layer of the platform, will be one of the cloud computing applications that have been widely used in data-intensive computing. The correct configuration of the configurable parameters of the MapReduce framework has an important effect on the performance of the calculation. However, when a heterogeneous Hadoop cluster is encountered, the user can only use the default configuration or manual configuration according to experience. Due to the large space available for parameter tuning, this often leads to poor performance due to misconfiguration. Aiming at the problems of various virtual machine platforms, this paper designs and implements a heterogeneous virtual machine management platform. On the basis of not changing the structure of the existing virtual machine management platform, the unified management and control of the existing mainstream virtual machine management platform and the balanced allocation of virtual resources are realized, and the extensible adaptation layer interface and driver components are also provided. Support for other heterogeneous virtual machine provisioning and management platforms. In order to solve the problem of Hadoop application on heterogeneous virtual machine platform, this paper presents a method of MapReduce online parameter automatic configuration based on reinforcement learning. This method uses off-line learning coarse-grained to create initialization strategy, on-line learning configures parameters according to the policy fine-grained, and iteratively updates the Q value table by trial and error method to make the configuration result close to optimal. The experimental results show that the proposed configuration method can effectively improve the performance of Hadoop, and can quickly iterate to achieve convergence, make full use of the machine resources running MapReduce tasks, and shorten the running time of the tasks.
【學(xué)位授予單位】：中南大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2013
【分類號】：TP302

【參考文獻(xiàn)】

相關(guān)期刊論文 前10條

1 張帆;李磊;楊成胡;陳麗珍;;基于Eucalyptus構(gòu)建私有云計(jì)算平臺[J];電信科學(xué);2011年11期

2 崔巍;李益發(fā);斯雪明;;基于Eucalyptus的基礎(chǔ)設(shè)施即服務(wù)云框架協(xié)議設(shè)計(jì)[J];電子與信息學(xué)報(bào);2012年07期

3 張倩;齊德昱;;面向服務(wù)的云制造協(xié)同設(shè)計(jì)平臺[J];華南理工大學(xué)學(xué)報(bào)(自然科學(xué)版);2011年12期

4 柴玉梅;景慧敏;;一種在多Agent系統(tǒng)中求帕累托效率解的方法[J];計(jì)算機(jī)工程與應(yīng)用;2010年22期

5 公偉;劉培玉;遲學(xué)芝;賈嫻;;云取證模型的構(gòu)建與分析[J];計(jì)算機(jī)工程;2012年11期

6 溫少君;陳俊杰;郭濤;;一種云平臺中優(yōu)化的虛擬機(jī)部署機(jī)制[J];計(jì)算機(jī)工程;2012年11期

7 柳香;李俊紅;段勝業(yè);;基于混沌PSO算法的Hadoop配置優(yōu)化[J];計(jì)算機(jī)工程;2012年11期

8 楊星;馬自堂;孫磊;;云環(huán)境下基于性能向量的虛擬機(jī)部署算法[J];計(jì)算機(jī)應(yīng)用;2012年01期

9 顧昊;錢曉俊;梁洪亮;;開源平臺下軟件管理技術(shù)的研究[J];計(jì)算機(jī)應(yīng)用研究;2007年08期

10 陳康;鄭緯民;;云計(jì)算:系統(tǒng)實(shí)例與研究現(xiàn)狀[J];軟件學(xué)報(bào);2009年05期

，

本文編號：2239011