本文選題：分布式存儲　切入點(diǎn)：HDFS　出處：《南京大學(xué)》2016年碩士論文

【摘要】：隨著大數(shù)據(jù)時代的到來,大規(guī)模數(shù)據(jù)存儲成為大數(shù)據(jù)的關(guān)鍵技術(shù)之一。分布式存儲系統(tǒng)大多部署在廉價的商用機(jī)器上,節(jié)點(diǎn)失效已經(jīng)成為一種常態(tài)。因此,如何構(gòu)建面向海量數(shù)據(jù)的可靠存儲機(jī)制成為相關(guān)技術(shù)研究的熱點(diǎn)。傳統(tǒng)基于多副本的容錯策略存在存儲開銷過高和容錯性差等問題,使其成為影響系統(tǒng)可擴(kuò)展能力的瓶頸。近年來,業(yè)界開始提出使用糾刪碼策略作為存儲系統(tǒng)的容錯機(jī)制,但糾刪碼策略存在數(shù)據(jù)修復(fù)帶寬消耗過大問題。為此,學(xué)界轉(zhuǎn)向?qū)诰W(wǎng)絡(luò)編碼的再生碼存儲策略開展研究。再生碼在修復(fù)時能達(dá)到最優(yōu)帶寬開銷,但再生碼計(jì)算開銷巨大等問題阻礙了其被廣泛應(yīng)用。另外,大多數(shù)存儲系統(tǒng)只使用固定的單一的編碼方法作為容錯策略,忽視所存儲文件本身的差異性,使得性能優(yōu)化還存在缺陷。針對上述問題,論文以構(gòu)建低冗余、高可用、高可靠的分布式存儲系統(tǒng)為目標(biāo),以基于HDFS的編碼存儲系統(tǒng)Cumulus為平臺,對基于再生碼的容錯分布式存儲及其性能優(yōu)化機(jī)制開展研究。主要工作包括以下兩個方面：1) 針對現(xiàn)有編碼方法的不足,結(jié)合存儲效率、訪問延遲、修復(fù)帶寬、計(jì)算復(fù)雜性等多維因素,提出了基于簡單再生碼的分布式容錯存儲方案,在此基礎(chǔ)上進(jìn)一步對簡單再生碼的退化讀修復(fù)機(jī)制進(jìn)行優(yōu)化設(shè)計(jì),并在Cumulus系統(tǒng)中實(shí)現(xiàn)了基于簡單再生碼的容錯存儲策略。實(shí)驗(yàn)結(jié)果表明,簡單再生碼在增加少量存儲開銷的基礎(chǔ)上,有效減少了修復(fù)開銷。2) 針對存儲系統(tǒng)中的文件加入生命周期和訪問頻率特征對文件訪問性能的影響問題,結(jié)合文件狀態(tài)和系統(tǒng)狀態(tài),提出了基于文件動態(tài)屬性的自適應(yīng)編碼機(jī)制。論文設(shè)計(jì)并實(shí)現(xiàn)了基于簡單再生碼的自適應(yīng)編碼模型。實(shí)驗(yàn)結(jié)果表明,基于文件動態(tài)屬性的自適應(yīng)編碼機(jī)制可有效提高分布式存儲系統(tǒng)的整體存儲效率,并降低修復(fù)代價。
[Abstract]:With the arrival of big data era, large-scale data storage has become one of the key technologies of big data.Distributed storage systems are mostly deployed on cheap commercial machines, and node failure has become the norm.Therefore, how to build a reliable storage mechanism for mass data has become a hot research topic.In recent years, erasure code strategy has been proposed as a fault-tolerant mechanism in storage systems. However, the erasure code strategy has the problem of excessive bandwidth consumption of data repair.Therefore, the academic circle turns to the research on the storage strategy of regenerative code based on network coding.The regenerative code can achieve the optimal bandwidth overhead when it is repaired, but it is widely used because of the huge computational overhead of the regenerated code.In addition, most storage systems only use a fixed single coding method as a fault-tolerant strategy, ignoring the differences of the stored files, which makes the performance optimization still have defects.Aiming at the above problems, this paper aims at building a distributed storage system with low redundancy, high availability and high reliability, and takes Cumulus, a coding storage system based on HDFS, as the platform.The fault-tolerant distributed storage based on regenerative code and its performance optimization mechanism are studied.The main work includes the following two aspects: (1) aiming at the shortcomings of the existing coding methods, combining the multi-dimensional factors such as storage efficiency, access delay, repair bandwidth, computational complexity and so on, a distributed fault-tolerant storage scheme based on simple regenerative code is proposed.On this basis, the degenerate read and repair mechanism of simple regenerative code is optimized, and the fault-tolerant storage strategy based on simple regenerative code is implemented in Cumulus system.Combining file state with system state, an adaptive encoding mechanism based on file dynamic attributes is proposed.An adaptive coding model based on simple regenerative codes is designed and implemented in this paper.Experimental results show that the adaptive coding mechanism based on file dynamic attributes can effectively improve the overall storage efficiency of distributed storage system and reduce the repair cost.
【學(xué)位授予單位】：南京大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2016
【分類號】：TP333

【參考文獻(xiàn)】

相關(guān)期刊論文 前2條

1 王永洲;茅蘇;;HDFS中的一種數(shù)據(jù)放置策略[J];計(jì)算機(jī)技術(shù)與發(fā)展;2013年05期

2 陳運(yùn)海;陳正亮;王欽輝;葉保留;;Cumulus:一個基于網(wǎng)絡(luò)編碼的分布式文件系統(tǒng)[J];微電子學(xué)與計(jì)算機(jī);2012年09期

，

本文編號：1722470