【摘要】：隨著當(dāng)今互聯(lián)網(wǎng)環(huán)境中信息呈爆炸式的增長(zhǎng)，人們對(duì)信息的存儲(chǔ)要求已經(jīng)變得越來(lái)越高。而大型互聯(lián)網(wǎng)公司內(nèi)部的分布式文件系統(tǒng)的集群規(guī)模更是急劇擴(kuò)張，存儲(chǔ)成本也在不斷上升。為了保證數(shù)據(jù)可靠性，現(xiàn)有分布式文件系統(tǒng)通常使用完全數(shù)據(jù)冗余的方法對(duì)文件進(jìn)行容錯(cuò)，其中采用最多的方法為三副本備份方法。但是三副本備份方法給系統(tǒng)帶來(lái)了相當(dāng)大的空間消耗，存儲(chǔ)效率低下，所以針對(duì)分布式文件系統(tǒng)的存儲(chǔ)效率進(jìn)行優(yōu)化越來(lái)越成為研究中的熱點(diǎn)。 糾刪碼技術(shù)通過(guò)對(duì)文件編碼生成校驗(yàn)來(lái)達(dá)到容錯(cuò)目的，能夠極大地提升分布式文件系統(tǒng)的存儲(chǔ)效率。但是由于糾刪碼方法需要對(duì)文件進(jìn)行編碼解碼操作，，并且在系統(tǒng)中僅保存了文件的一份副本，對(duì)系統(tǒng)性能帶來(lái)了影響。因此針對(duì)此問(wèn)題，提出一種基于冷熱數(shù)據(jù)區(qū)分的混合存儲(chǔ)方案，在提高分布式文件系統(tǒng)存儲(chǔ)效率的同時(shí)，將糾刪碼技術(shù)對(duì)系統(tǒng)性能的損耗降低到最小。 所提出的方案結(jié)合了三副本備份方法和糾刪碼方法的優(yōu)點(diǎn)，將新寫入的數(shù)據(jù)以三副本備份方法存儲(chǔ)，保證系統(tǒng)性能，當(dāng)數(shù)據(jù)變?yōu)槔鋽?shù)據(jù)時(shí)，對(duì)文件進(jìn)行編碼轉(zhuǎn)換，并減少副本數(shù)，節(jié)約存儲(chǔ)空間。在編碼過(guò)程中系統(tǒng)參考了RAID（RedundantArray ofIndependent Disk）中“條帶化”的思想以一個(gè)條帶做為編碼單元。 測(cè)試結(jié)果表明，此存儲(chǔ)效率優(yōu)化方案相對(duì)于三副本備份方法在存儲(chǔ)效率方面提升了約25%，并且此方案將糾刪碼技術(shù)所帶來(lái)的系統(tǒng)性能損耗由50%降低至5%以內(nèi)，其性能表現(xiàn)與三副本備份方法基本一致。測(cè)試說(shuō)明了基于冷熱數(shù)據(jù)區(qū)分的存儲(chǔ)效率優(yōu)化方案確實(shí)集合了多種方案的優(yōu)勢(shì)，并在存儲(chǔ)成本、數(shù)據(jù)可靠性和系統(tǒng)性能之間達(dá)到了平衡。
[Abstract]:With the explosive growth of information in today's Internet environment, the storage requirements of information have become higher and higher. The cluster size of distributed file system within large Internet companies is expanding rapidly, and the storage cost is also rising. In order to ensure the reliability of data, the existing distributed file systems usually use the method of complete data redundancy to fault-tolerant the files, among which the most common method is the three-copy backup method. However, the three-copy backup method brings a lot of space consumption and low storage efficiency to the system, so the optimization of storage efficiency of distributed file system has become a hot research topic. Erasure code technology can greatly improve the storage efficiency of distributed file system by generating and checking the file coding to achieve the purpose of fault tolerance. However, because the erasure code method needs to encode and decode the file, and only one copy of the file is saved in the system, which has an impact on the performance of the system. Therefore, in order to solve this problem, a hybrid storage scheme based on cold and hot data differentiation is proposed, which not only improves the storage efficiency of distributed file system, but also minimizes the loss of erasure code technology to system performance. The proposed scheme combines the advantages of the three-copy backup method and the erasure code method, and stores the newly written data as a three-copy backup method to ensure the system performance. When the data becomes cold data, the file is encoded and converted. And reduce the number of copies, save storage space. In the process of coding, a stripe is used as the coding unit, which refers to the idea of striping in RAID (RedundantArray ofIndependent Disk). The test results show that the storage efficiency optimization scheme improves the storage efficiency by about 25% compared with the three-copy backup method, and this scheme reduces the system performance loss caused by erasure code technology from 50% to less than 5%. Its performance is basically consistent with the three-copy backup method. The test shows that the storage efficiency optimization scheme based on cold and hot data differentiation does combine the advantages of various schemes and strike a balance between storage cost, data reliability and system performance.
【學(xué)位授予單位】：華中科技大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2013
【分類號(hào)】：TP333

【參考文獻(xiàn)】

相關(guān)期刊論文 前1條

1 羅象宏;舒繼武;;存儲(chǔ)系統(tǒng)中的糾刪碼研究綜述[J];計(jì)算機(jī)研究與發(fā)展;2012年01期

相關(guān)博士學(xué)位論文 前1條

1 李明強(qiáng);磁盤陣列的糾刪碼技術(shù)研究[D];清華大學(xué);2011年

本文編號(hào)：2479298