本文關(guān)鍵詞：Hadoop框架的擴展和性能調(diào)優(yōu)　出處：《西安建筑科技大學》2012年碩士論文　論文類型：學位論文

  更多相關(guān)文章： 云計算 網(wǎng)格計算 LSF Hadoop MapReduce

【摘要】：云計算作為一種新的概念在2007年成為了人們熱議的話題，在隨后的幾年內(nèi)云計算得到了快速的發(fā)展。從計算模式來看，云計算、分布式計算和網(wǎng)格計算有很多相似之處，深入研究云計算產(chǎn)生的背景就可以看出，云計算是在分布式計算和網(wǎng)格計算的基礎(chǔ)之上發(fā)展起來的。以前的分布式計算和網(wǎng)格計算主要用于科學研究方面，隨著互聯(lián)網(wǎng)的迅速發(fā)展，分布式計算和網(wǎng)格計算的思想逐漸演化為一種更適合商用的計算模式-云計算。 論文首先介紹了云計算與網(wǎng)格計算的相關(guān)背景知識，并分析了兩者之間的區(qū)別，然后對云計算平臺Hadoop核心組成MapReduce、HDFS(Hadoop Distributed FileSystem)和Hbase等的關(guān)鍵技術(shù)進行詳細的分析與研究[1]。接著詳細介紹了LSF(Load Sharing Facility)系統(tǒng)的架構(gòu)組成，包括LSF base和LSF batch兩部分，并對LSF的作業(yè)執(zhí)行流程和系統(tǒng)負載均衡進行了深入細致的分析。 論文在對Hadoop系統(tǒng)深入研究分析之后，發(fā)現(xiàn)Hadoop在面對企業(yè)級應(yīng)用時有3大不足，分別是單點故障、調(diào)度算法單一、異構(gòu)平臺兼容性差[2]。針對這幾點不足，論文對Hadoop系統(tǒng)與LSF系統(tǒng)進行了關(guān)聯(lián)性整合，形成一個新的系統(tǒng)LSH(LoadShare Hadoop)。系統(tǒng)整合主要有兩大結(jié)合點，第一，，將LSF的作業(yè)控制機制LIM(Load Information Manager)、RES (Remote Execution Server)和SBD(sbatch,一個守護進程)加入到Hadoop系統(tǒng)的HDFS層與MapReduce層之間；第二，LSF的master節(jié)點與HDFS的NameNode之間通過開放接口共享信息。整合后的系統(tǒng)LSH有效地防止了Hadoop系統(tǒng)的單點故障問題，也解決了Hadoop調(diào)度算法單一的問題和Hadoop對異構(gòu)平臺的兼容性問題。 論文最后針對整合后的系統(tǒng)LSH和原生態(tài)的Hadoop系統(tǒng)設(shè)計了不同的實驗，分別來驗證兩系統(tǒng)對單點故障的處理、差異性作業(yè)的性能和異構(gòu)平臺的適應(yīng)性方面的表現(xiàn)，結(jié)果證明LSH系統(tǒng)完全彌補了原生Hadoop的不足，LSH是能夠適應(yīng)企業(yè)級的應(yīng)用。
[Abstract]:Cloud computing as a new concept in 2007 has become a hot topic, cloud computing has been rapid development in the following years. From the perspective of computing mode, cloud computing. There are many similarities between distributed computing and grid computing. Cloud computing is developed on the basis of distributed computing and grid computing. The former distributed computing and grid computing are mainly used in scientific research, with the rapid development of the Internet. The idea of distributed computing and grid computing has evolved into a more commercial computing model-cloud computing. Firstly, this paper introduces the background knowledge of cloud computing and grid computing, and analyzes the difference between them. Then, the Hadoop core of cloud computing platform is composed of MapReduce. The key technologies of HDFS(Hadoop Distributed File system and Hbase are analyzed and studied in detail. [1. Then the architecture of the LSF(Load Sharing availability) system is introduced in detail, including two parts: LSF base and LSF batch. The job execution flow and system load balance of LSF are analyzed in detail. After deeply studying and analyzing Hadoop system, it is found that Hadoop has three shortcomings in facing enterprise application, namely, single point fault, single scheduling algorithm and poor compatibility of heterogeneous platform. [2]. Aiming at these shortcomings, this paper integrates the Hadoop system and the LSF system. To form a new system LSH(LoadShare Hadoop. System integration has two main points of convergence, first. LSF's job control mechanism, LIM(Load Information Manager. RES remote Execution Server) and SBD(sbatch. A daemon is added between the HDFS layer and the MapReduce layer of the Hadoop system; Number two. The master node of LSF and the NameNode of HDFS share information through open interface. The integrated system LSH effectively prevents the single point of failure of Hadoop system. It also solves the problem of single Hadoop scheduling algorithm and compatibility of Hadoop to heterogeneous platforms. At the end of the paper, different experiments are designed for the integrated system LSH and the original Hadoop system, respectively, to verify the two systems to deal with the single point of failure. The performance of different jobs and the adaptability of heterogeneous platforms show that the LSH system can fully compensate for the deficiency of native Hadoop and is able to adapt to enterprise-level applications.
【學位授予單位】：西安建筑科技大學
【學位級別】：碩士
【學位授予年份】：2012
【分類號】：TP338.8

【參考文獻】

相關(guān)期刊論文 前3條

1 李鑫;張鵬;;Hadoop集群公平調(diào)度算法的改進與實現(xiàn)[J];電腦知識與技術(shù);2012年01期

2 李成華;張新訪;金海;向文;;MapReduce:新型的分布式并行計算編程模型[J];計算機工程與科學;2011年03期

3 孫廣中;肖鋒;熊曦;;MapReduce模型的調(diào)度及容錯機制研究[J];微電子學與計算機;2007年09期

相關(guān)碩士學位論文 前6條

1 陳艷金;MapReduce模型在Hadoop平臺下實現(xiàn)作業(yè)調(diào)度算法的研究和改進[D];華南理工大學;2011年

2 徐文強;基于HDFS的云存儲系統(tǒng)研究[D];上海交通大學;2011年

3 張文峰;基于MapReduce模型的分布式計算平臺的原理與設(shè)計[D];華中科技大學;2010年

4 杜志源;基于OGSA的教育資源共享研究[D];西安電子科技大學;2007年

5 夏yN;Hadoop平臺下的作業(yè)調(diào)度算法研究與改進[D];華南理工大學;2010年

6 張密密;MapReduce模型在Hadoop實現(xiàn)中的性能分析及改進優(yōu)化[D];電子科技大學;2010年

本文編號：1422823