發(fā)布時(shí)間：2018-07-05 18:44

  本文選題：HDFS + 分布式文件系統(tǒng)　； 參考：《北京郵電大學(xué)》2012年碩士論文

【摘要】：隨著信息技術(shù)的迅猛發(fā)展,數(shù)據(jù)的存儲(chǔ)占有越來越重要的地位。在數(shù)據(jù)呈指數(shù)級(jí)增長(zhǎng)的今天,由于容量、價(jià)格和安全性的限制,本地的存儲(chǔ)已經(jīng)逐漸力不從心。這使得由分布式文件系統(tǒng)所支持的云存儲(chǔ)的應(yīng)用越來越廣泛,其中Hadoop distributed file system(HDFS)因其強(qiáng)大的容錯(cuò)能力和可擴(kuò)展性而得到了人們的廣泛關(guān)注。但由于其設(shè)計(jì)是仿照Google File System(GFS),因此其只是對(duì)搜索引擎應(yīng)用的支持比較好,要想應(yīng)用于一般性的分布式存儲(chǔ),還需要一些研究和改進(jìn)。 在搜索引擎的應(yīng)用中,文件大部分都是以大文件的形式存在的。而在一般性的存儲(chǔ)中,文件的大小是多種多樣的。而且由于HDFS單一名字節(jié)點(diǎn)(Namenode)的性能瓶頸問題,在文件分塊太多的情況下,其對(duì)數(shù)據(jù)的訪問會(huì)變得比較糟糕。因此,盡管HDFS具有很多先進(jìn)的特性,但其設(shè)計(jì)的初衷就決定了其不是一個(gè)普適的分布式文件系統(tǒng),而只能支持很有限的應(yīng)用。 本文的目的是設(shè)計(jì)和實(shí)現(xiàn)一個(gè)用于云存儲(chǔ)的普適的分布式文件系統(tǒng)。首先提出了一種多Namenode的分布式文件系統(tǒng)的架構(gòu)。文件的元數(shù)據(jù)由多個(gè)Namenode分布式的存儲(chǔ),并且名字節(jié)點(diǎn)只存儲(chǔ)file到block的映射,而block的位置信息由數(shù)據(jù)節(jié)點(diǎn)管理者(DatanodeManager)進(jìn)行存儲(chǔ),從而降低了名字節(jié)點(diǎn)的負(fù)載。 然后,本文主要針對(duì)Datanode集群部分的實(shí)現(xiàn)方案和關(guān)鍵算法進(jìn)行了論述。其中,本文對(duì)Datanode集群中的block分塊策略進(jìn)行了重新設(shè)計(jì)和實(shí)現(xiàn)。該策略中,數(shù)據(jù)塊的大小有多個(gè)分塊的因子供選擇,系統(tǒng)根據(jù)具體的應(yīng)用類型和文件的屬性等信息靈活的對(duì)文件進(jìn)行分塊,從而保證系統(tǒng)能夠?qū)τ谠破脚_(tái)上的各種應(yīng)用都有良好的訪問性能。
[Abstract]:With the rapid development of information technology, data storage plays an increasingly important role. In today's exponential growth of data, local storage has been overwhelmed by capacity, price and security constraints. This makes the application of cloud storage supported by distributed file system more and more extensive, among which Hadoop distributed file system (HDFS) has been paid more and more attention because of its strong fault tolerance and extensibility. However, because its design is modelled on Google File system (GFS), it only supports search engine applications better. If it is to be applied to general distributed storage, it still needs some research and improvement. In the application of search engine, most files exist in the form of large files. In general storage, the size of the file is varied. Moreover, because of the performance bottleneck of HDFS single name node (Namenode), the access to data becomes worse when there are too many file blocks. Therefore, although HDFS has many advanced features, the original intention of its design determines that HDFS is not a universal distributed file system, but can only support very limited applications. The purpose of this paper is to design and implement a pervasive distributed file system for cloud storage. Firstly, a distributed file system architecture with multiple Namenode is proposed. The metadata of the file is stored by several Namenodes distributed, and the name node only stores the mapping of file to block, while the location information of block is stored by the data node manager, which reduces the load of the name node. Then, this paper mainly discusses the implementation scheme and key algorithms of the DataNode cluster. Among them, this paper redesigns and implements the block partitioning strategy in the DataNode cluster. In this strategy, the size of the data block has a number of block factors to choose, and the system can block the file flexibly according to the specific application type and file attributes, etc. In order to ensure that the system can have good access to various applications on the cloud platform.
【學(xué)位授予單位】：北京郵電大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2012
【分類號(hào)】：TP333

【參考文獻(xiàn)】

相關(guān)期刊論文 前5條

1 李東升,李春江,肖儂,王意潔,盧錫城;數(shù)據(jù)網(wǎng)格環(huán)境下一種動(dòng)態(tài)自適應(yīng)的副本定位方法[J];計(jì)算機(jī)研究與發(fā)展;2003年12期

2 王雪濤;劉偉杰;;分布式文件系統(tǒng)[J];科技信息(學(xué)術(shù)研究);2006年11期

3 王正;羅萬明;閻保平;;并行下載最優(yōu)機(jī)制[J];軟件學(xué)報(bào);2009年08期

4 彭安鑫;石磊;;云計(jì)算技術(shù)概述[J];網(wǎng)絡(luò)安全技術(shù)與應(yīng)用;2011年06期

5 陳俊杰,張武生,沈美明,鄭緯民;文件分配問題的一種動(dòng)態(tài)解決算法[J];小型微型計(jì)算機(jī)系統(tǒng);2004年07期

相關(guān)碩士學(xué)位論文 前1條

1 黃曉云;基于HDFS的云存儲(chǔ)服務(wù)系統(tǒng)研究[D];大連海事大學(xué);2010年

，

本文編號(hào)：2101352