本文選題：異構(gòu)集群　切入點：計算能力　出處：《華中科技大學(xué)》2016年碩士論文

【摘要】：隨著基于Hadoop平臺的大數(shù)據(jù)技術(shù)不斷發(fā)展和實踐的深入,Hadoop YARN(Yet Anouther Resource Negotiator)資源調(diào)度策略在異構(gòu)集群中的不適用性越發(fā)明顯。一方面,YARN資源調(diào)度器無法根據(jù)節(jié)點的計算能力動態(tài)調(diào)整節(jié)點承擔的任務(wù)份額,導(dǎo)致異構(gòu)集群中優(yōu)勢節(jié)點的計算資源浪費、系統(tǒng)性能沒有充分發(fā)揮;另一方面,現(xiàn)有的資源調(diào)度策略始終為作業(yè)靜態(tài)地分配統(tǒng)一規(guī)格的資源容器,未考慮作業(yè)執(zhí)行的不同階段資源需求的差異性,易產(chǎn)生大量資源碎片,從而導(dǎo)致系統(tǒng)資源利用率降低,整體性能下降�；谝陨蠁栴},在深入分析YARN架構(gòu)及其資源調(diào)度機制的基礎(chǔ)上提出了作業(yè)的自適應(yīng)資源調(diào)度策略:首先,監(jiān)控服務(wù)器對集群所有執(zhí)行節(jié)點和提交的作業(yè)進行多項性能相關(guān)信息的監(jiān)控;其次,利用采集的實時監(jiān)控數(shù)據(jù)建模、量化集群節(jié)點的綜合計算能力;最后,集群主節(jié)點結(jié)合實時節(jié)點性能監(jiān)控信息和作業(yè)性能監(jiān)控信息啟動基于相似度評估的動態(tài)資源調(diào)度方案。優(yōu)化后的系統(tǒng)能夠有效識別集群節(jié)點的執(zhí)行能力差異,并根據(jù)作業(yè)任務(wù)的實時需求進行細粒度的動態(tài)資源調(diào)度,在完善YARN現(xiàn)有調(diào)度語義的同時,可作為子級資源調(diào)度方案架構(gòu)在上層調(diào)度器下。搭建Hadoop2.0和Ganglia綜合實驗平臺,對上述作業(yè)的自適應(yīng)資源調(diào)度策略進行實現(xiàn),并基于大數(shù)據(jù)典型CPU密集型作業(yè)和I/O密集型作業(yè)進行性能測試。實驗結(jié)果表明,作業(yè)的自適應(yīng)資源調(diào)度策略能夠有效增加集群并發(fā)度、縮短作業(yè)執(zhí)行時間、提升系統(tǒng)資源利用率。
[Abstract]:With the continuous development and practice of big data technology based on Hadoop platform, the inapplicability of Hadoop YARN(Yet Anouther Resource negotiator resource scheduling strategy in heterogeneous clusters becomes more and more obvious.On the one hand, the YARN resource scheduler can not dynamically adjust the task share of the node according to the computing power of the node, which leads to the waste of computing resources of the dominant node in the heterogeneous cluster, and the system performance is not given full play.The existing resource scheduling strategy always assigns uniform resource container to the job statically, and does not consider the difference of resource requirements in different stages of job execution, which is easy to produce a large number of resource fragments, which leads to the decrease of system resource utilization.Overall performance decline.Based on the above problems, an adaptive resource scheduling strategy for jobs is proposed on the basis of in-depth analysis of the YARN architecture and its resource scheduling mechanism.The monitoring server monitors the performance related information of all the execution nodes and the jobs submitted by the cluster. Secondly, using the collected real-time monitoring data modeling, quantifies the comprehensive computing ability of the cluster nodes.The dynamic resource scheduling scheme based on similarity evaluation is initiated by cluster master node combining real-time node performance monitoring information and job performance monitoring information.The optimized system can effectively identify the difference of execution ability of cluster nodes, and carry out fine-grained dynamic resource scheduling according to the real-time requirements of job tasks.It can be used as a sublevel resource scheduling scheme architecture under the upper scheduler.A comprehensive experimental platform of Hadoop2.0 and Ganglia was built to implement the adaptive resource scheduling strategy of the above jobs and to test the performance based on big data typical CPU intensive jobs and I / O intensive jobs.The experimental results show that the adaptive resource scheduling strategy can effectively increase the concurrency degree of the cluster, shorten the job execution time and improve the system resource utilization.
【學(xué)位授予單位】：華中科技大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2016
【分類號】：TP311.13

【參考文獻】

相關(guān)期刊論文 前5條

1 楊倩茹;黃夢醒;萬兵;;一種引入內(nèi)存平衡的Hadoop平臺作業(yè)調(diào)度算法[J];小型微型計算機系統(tǒng);2014年12期

2 劉衛(wèi)寧;高龍;;異構(gòu)云中面向集群負載均衡的任務(wù)調(diào)度策略[J];計算機應(yīng)用;2013年08期

3 楊卓犖;;基于YARN構(gòu)建多功能分布式集群[J];程序員;2013年06期

4 黃，

本文編號：1701884