【摘要】：隨著科技的不斷發(fā)展以及全球數(shù)據(jù)量的激增，云存儲(chǔ)與云計(jì)算是未來的發(fā)展趨勢(shì)，傳統(tǒng)數(shù)據(jù)庫對(duì)數(shù)據(jù)的處理已經(jīng)越來越不能滿足個(gè)人與企業(yè)用戶的要求，對(duì)于海量數(shù)據(jù)，業(yè)界大數(shù)據(jù)存儲(chǔ)及分布式處理系統(tǒng)最有代表性的就是Hadoop。Hadoop在最近幾年迅猛發(fā)展，它是一個(gè)具有可靠性、高效性、可伸縮性的能夠?qū)Υ罅繑?shù)據(jù)進(jìn)行分布式處理的開源軟件框架。 由于設(shè)計(jì)Hadoop之初是假設(shè)集群所有機(jī)器都是同構(gòu)的，而現(xiàn)實(shí)中，Hadoop集群是有許多廉價(jià)機(jī)器組成，這就導(dǎo)致了集群中的節(jié)點(diǎn)計(jì)算能力的差異以及節(jié)點(diǎn)容易失效的問題，雖然Hadoop為了防止計(jì)算任務(wù)和數(shù)據(jù)存儲(chǔ)可能會(huì)失敗而維護(hù)了多個(gè)數(shù)據(jù)副本，以提高集群的容錯(cuò)能力與可靠性。但是在預(yù)測(cè)節(jié)點(diǎn)失效與數(shù)據(jù)副本放置以及任務(wù)調(diào)度上仍然需要完善和改進(jìn)。 為了提高Hadoop集群的魯棒性，本文在不同性能的節(jié)點(diǎn)執(zhí)行任務(wù)效率的差異下對(duì)其魯棒性進(jìn)行了優(yōu)化，研究的主要內(nèi)容如下： （1）針對(duì)Hadoop在任務(wù)節(jié)點(diǎn)的選取與數(shù)據(jù)副本放置時(shí)未考慮節(jié)點(diǎn)未來可能會(huì)失效的問題，提出了Hadoop節(jié)點(diǎn)故障預(yù)測(cè)模型，對(duì)集群中的節(jié)點(diǎn)進(jìn)行了故障率預(yù)測(cè)。 （2）通過節(jié)點(diǎn)故障預(yù)測(cè)模型，對(duì)于Hadoop任務(wù)調(diào)度進(jìn)行了優(yōu)化以及提出了關(guān)于數(shù)據(jù)副本放置的節(jié)點(diǎn)選擇策略算法。解決了默認(rèn)算法未考慮節(jié)點(diǎn)異構(gòu)性而造成的計(jì)算能力差異的問題，提高了集群的魯棒性。 （3）對(duì)于集群中執(zhí)行任務(wù)次數(shù)較少以及通過節(jié)點(diǎn)故障預(yù)測(cè)模型判斷出高故障率的節(jié)點(diǎn)，建立了休眠機(jī)制，，解決了該類節(jié)點(diǎn)的處置問題。 （4）通過搭建Hadoop集群驗(yàn)證了故障預(yù)測(cè)模型在集群計(jì)算效率約束下的有效性，本文所提出的方法提高了Hadoop集群的魯棒性。
[Abstract]:With the development of science and technology, cloud storage and cloud computing are the development trend in the future. The traditional data processing can not meet the needs of individuals and enterprise users. Big data storage and distributed processing system is the most representative of the development of Hadoop.Hadoop in recent years, it is a reliable, efficient, scalable open source software framework for distributed processing of a large number of data. Since the Hadoop was designed on the assumption that all machines in the cluster are isomorphic, in reality, the Hadoop cluster is made up of many cheap machines, which leads to the difference in the computing power of the nodes in the cluster and the problem that the nodes are prone to failure. Although Hadoop maintains multiple copies of data in order to prevent computing tasks and data storage from failing to improve the fault tolerance and reliability of the cluster. However, the prediction of node failure, data copy placement and task scheduling still needs improvement. In order to improve the robustness of Hadoop cluster, this paper optimizes the robustness of different performance nodes under different task efficiency. The main contents of the research are as follows: (1) aiming at the problem that Hadoop does not consider the possible future failure of Hadoop in the task node selection and data copy placement, a Hadoop node fault prediction model is proposed. The failure rate of the nodes in the cluster is predicted. (2) based on the node fault prediction model, the Hadoop task scheduling is optimized and a node selection strategy algorithm for data replica placement is proposed. It solves the problem that the default algorithm does not take into account the heterogeneity of nodes and improves the robustness of the cluster. (3) for nodes with low number of task execution and high failure rate determined by node fault prediction model, a dormancy mechanism is established to solve the problem of dealing with this kind of nodes. (4) the Hadoop cluster is built to verify the effectiveness of the fault prediction model under the constraint of cluster computing efficiency. The proposed method improves the robustness of the Hadoop cluster.
【學(xué)位授予單位】：遼寧大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2014
【分類號(hào)】：TP311.13;TP333

【相似文獻(xiàn)】

相關(guān)期刊論文 前10條

1 李強(qiáng);;一種基于診斷融合技術(shù)的傳感器節(jié)點(diǎn)故障診斷[J];科技通報(bào);2013年04期

2 孫德印;孫鐵強(qiáng);;彩電檢修的新方法——“節(jié)點(diǎn)”分析法[J];家電檢修技術(shù);2003年08期

3 李斌,張陽;節(jié)點(diǎn)與屏幕墻[J];現(xiàn)代電視技術(shù);2004年06期

4 張啟忠;席旭剛;;基于粗糙集分解的傳感器網(wǎng)絡(luò)節(jié)點(diǎn)故障診斷方法[J];杭州電子科技大學(xué)學(xué)報(bào);2010年04期

5 劉茂立;鄧忠良;;基于選擇關(guān)鍵節(jié)點(diǎn)的網(wǎng)絡(luò)易損性評(píng)估法[J];艦船電子工程;2011年01期

6 高志鵬;黃日茂;陳穎慧;芮蘭蘭;;一種無線傳感器網(wǎng)絡(luò)節(jié)點(diǎn)故障的檢測(cè)方法(英文)[J];中國通信;2011年01期

7 張R

本文編號(hào)：2395026