本文關(guān)鍵詞：基于云存儲(chǔ)的數(shù)據(jù)流處理技術(shù)的研究　出處：《武漢理工大學(xué)》2013年碩士論文　論文類型：學(xué)位論文

  更多相關(guān)文章： 云存儲(chǔ) 數(shù)據(jù)流處理 HDFS Map/Reduce

【摘要】：自2006年Google提出云計(jì)算概念以來,云計(jì)算從備受業(yè)界質(zhì)疑的概念炒作成為如今越來越成熟的技術(shù)服務(wù)形態(tài)。在云計(jì)算提供的眾多服務(wù)類型里,存儲(chǔ)服務(wù)成為我們最為直接使用的一種云計(jì)算服務(wù),并發(fā)展成為業(yè)界獨(dú)立研究的領(lǐng)域,目前眾多的IT巨頭都在云存儲(chǔ)領(lǐng)域進(jìn)行布局。云存儲(chǔ)是為大數(shù)據(jù)時(shí)代而生的,如何更加高效、快速、安全的進(jìn)行海量數(shù)據(jù)的存儲(chǔ)、管理和處理,仍然是吸引眾多IT人士不斷研究的課題。如今,在云存儲(chǔ)的后端,Hadoop作為最適合大數(shù)據(jù)處理的開源技術(shù),正被廣泛的研究和使用。但由于Hadoop的興起時(shí)間不長(zhǎng),其還存在著一些設(shè)計(jì)缺陷,并且由于眾多云存儲(chǔ)服務(wù)提供商會(huì)根據(jù)服務(wù)的類別和數(shù)據(jù)中心的實(shí)際情況,需要對(duì)Hadoop進(jìn)行改進(jìn),以便提供更好的服務(wù)。 本文的研究?jī)?nèi)容包括云存儲(chǔ)中數(shù)據(jù)流處理關(guān)鍵技術(shù)Hadoop,Hadoop是把數(shù)據(jù)以流的方式來進(jìn)行處理的。通過對(duì)Hadoop平臺(tái)中核心組件HDFS分布式文件系統(tǒng)的框架和執(zhí)行流程的研究,針對(duì)其單一主控節(jié)點(diǎn)NameNode的設(shè)計(jì)缺陷,提出了一種主控節(jié)點(diǎn)壓力分解的方法,在可接受的性能損失范圍內(nèi),減輕了HDFS架構(gòu)中單一主控節(jié)點(diǎn)的訪問壓力,對(duì)系統(tǒng)架構(gòu)做出了一定的修改,使其在總體上能夠承擔(dān)更多的訪問請(qǐng)求,并且降低了單一節(jié)點(diǎn)在過量負(fù)載時(shí)造成的不穩(wěn)定性甚至是崩潰的風(fēng)險(xiǎn),使系統(tǒng)的健壯性進(jìn)一步提高。另外,本文對(duì)HDFS的元數(shù)據(jù)信息進(jìn)行二次備份設(shè)計(jì),進(jìn)一步提高了系統(tǒng)的可靠性。 本文還對(duì)Hadoop的另一核心組件Map/Reduce的數(shù)據(jù)流處理機(jī)制進(jìn)行研究,針對(duì)其過于消耗資源的缺點(diǎn),提出了一種在特定情況下能夠有效的降低Map/Reduce的資源消耗的優(yōu)化方式。本文針對(duì)這些特殊情況下,對(duì)元數(shù)據(jù)的數(shù)據(jù)結(jié)構(gòu)進(jìn)行改進(jìn),使Map/Reduce在進(jìn)行數(shù)據(jù)流處理之前能夠先從HDFS獲得元數(shù)據(jù)信息,進(jìn)行數(shù)據(jù)塊的精準(zhǔn)定位,過濾掉不必要的數(shù)據(jù)處理,使HDFS對(duì)Map/Reduce的支持進(jìn)一步提高,能夠有效的降低數(shù)據(jù)處理過程中的資源消耗,避免了資源的浪費(fèi)。 在本文的最后,通過多次實(shí)驗(yàn),使優(yōu)化后的系統(tǒng)與原始架構(gòu)中數(shù)據(jù)處理情況進(jìn)行比較。實(shí)驗(yàn)數(shù)據(jù)表明,改進(jìn)后的系統(tǒng)在平衡資源消耗以及負(fù)載壓力情況下,能夠達(dá)到我們預(yù)期的結(jié)果。 本文得到國(guó)家自然科學(xué)基金項(xiàng)目(批準(zhǔn)號(hào)：60970064)的資助。
[Abstract]:Since Google put forward the concept of cloud computing in 2006, cloud computing has become a more and more mature technology service form from the concept that has been questioned by the industry. Storage service has become the most direct use of cloud computing services, and has developed into an independent field of research in the industry. At present, a large number of IT giants are in the cloud storage field layout. Cloud storage is for the era of big data, how to more efficient, fast, safe storage, management and processing of massive data. Today, Hadoop on the back end of cloud storage is best suited to big data's open source technology. Is being widely studied and used, but because the rise of Hadoop time is not long, it also has some design defects. And because many cloud storage service providers need to improve Hadoop to provide better service according to the type of service and the actual situation of data center. The research content of this paper includes Hadoop, the key technology of data stream processing in cloud storage. Hadoop deals with the data in the way of stream. Through the research of the framework and execution flow of HDFS distributed file system, the core component of Hadoop platform. Aiming at the design defect of NameNode, a method of pressure decomposition is proposed, which is in the range of acceptable performance loss. It reduces the access pressure of the single master node in the HDFS architecture and makes some modifications to the system architecture so that it can take on more access requests on the whole. It also reduces the risk of instability or even crash caused by a single node in excess load, and further improves the robustness of the system. In addition, this paper designs the secondary backup of HDFS metadata information. The reliability of the system is further improved. This paper also studies the data flow processing mechanism of Map/Reduce, another core component of Hadoop, aiming at its shortcomings of consuming too much resources. This paper proposes an optimization method that can effectively reduce the resource consumption of Map/Reduce under certain circumstances. This paper improves the data structure of metadata under these special circumstances. The Map/Reduce can obtain metadata information from HDFS before processing data flow, locate the data block accurately and filter out unnecessary data processing. The support of HDFS to Map/Reduce can be further improved, which can effectively reduce the resource consumption in the process of data processing and avoid the waste of resources. At the end of this paper, through many experiments, the optimized system is compared with the data processing in the original architecture. The experimental data show that the improved system is balanced in the case of resource consumption and load pressure. Be able to achieve the desired results. This paper is supported by the National Natural Science Foundation of China (Grant No.: 60970064).
【學(xué)位授予單位】：武漢理工大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2013
【分類號(hào)】：TP333

【參考文獻(xiàn)】

相關(guān)期刊論文 前10條

1 蒙安泰;;分布式文件系統(tǒng)中元數(shù)據(jù)管理機(jī)制的研究[J];電腦知識(shí)與技術(shù);2011年35期

2 錢宏蕊;;云存儲(chǔ)技術(shù)發(fā)展及應(yīng)用[J];電信工程技術(shù)與標(biāo)準(zhǔn)化;2012年04期

3 吳海佳;陳衛(wèi)衛(wèi);劉鵬;董繼光;;云存儲(chǔ)系統(tǒng)中基于更新日志的元數(shù)據(jù)緩存同步策略[J];電信科學(xué);2011年09期

4 劉正偉;文中領(lǐng);張海濤;;云計(jì)算和云數(shù)據(jù)管理技術(shù)[J];計(jì)算機(jī)研究與發(fā)展;2012年S1期

5 任宇寧;;云計(jì)算時(shí)代的存儲(chǔ)技術(shù)——云存儲(chǔ)[J];科技傳播;2012年03期

6 冀素琴;石洪波;衛(wèi)潔;;基于Map Reduce的Bagging貝葉斯文本分類[J];計(jì)算機(jī)工程;2012年16期

7 鄧鵬;李枚毅;何誠(chéng);;Namenode單點(diǎn)故障解決方案研究[J];計(jì)算機(jī)工程;2012年21期

8 傅穎勛;羅圣美;舒繼武;;安全云存儲(chǔ)系統(tǒng)與關(guān)鍵技術(shù)綜述[J];計(jì)算機(jī)研究與發(fā)展;2013年01期

9 方少卿;周劍;張明新;;基于Map/Reduce的改進(jìn)選擇算法在云計(jì)算的Web數(shù)據(jù)挖掘中的研究[J];計(jì)算機(jī)應(yīng)用研究;2013年02期

10 徐小龍;周靜嵐;楊庚;;一種基于數(shù)據(jù)分割與分級(jí)的云存儲(chǔ)數(shù)據(jù)隱私保護(hù)機(jī)制[J];計(jì)算機(jī)科學(xué);2013年02期

相關(guān)碩士學(xué)位論文 前2條

1 葉雄杰;基于云存儲(chǔ)的移動(dòng)視頻監(jiān)控系統(tǒng)研究[D];廣東工業(yè)大學(xué);2011年

2 李寬;基于HDFS的分布式Namenode節(jié)點(diǎn)模型的研究[D];華南理工大學(xué);2011年

，

本文編號(hào)：1374367