【摘要】：隨著互聯(lián)網(wǎng)在生活中的廣泛應(yīng)用,網(wǎng)絡(luò)中的數(shù)據(jù)量呈指數(shù)級增長,IT行業(yè)已經(jīng)從計算時代進(jìn)入到存儲時代。目前,許多數(shù)據(jù)中心都面臨著系統(tǒng)能耗開銷巨大、存儲資源利用率較低以及存儲成本不斷上升等問題。有數(shù)據(jù)表明,存儲的各種開支已經(jīng)占數(shù)據(jù)中心總成本的30%-70%。而日常產(chǎn)生的數(shù)據(jù)中,經(jīng)常使用的數(shù)據(jù)(熱數(shù)據(jù))只占到整體數(shù)據(jù)量的20%而極少使用或幾乎不用的數(shù)據(jù)(冷數(shù)據(jù))高達(dá)80%。在這種情況下,根據(jù)數(shù)據(jù)冷熱特征進(jìn)行分級存儲,發(fā)揮各種存儲設(shè)備優(yōu)點,降低總存儲成本,有十分廣闊的研究和應(yīng)用前景。對比目前存儲介質(zhì)技術(shù)的發(fā)展,光盤的單位容量價格遠(yuǎn)遠(yuǎn)低于磁盤,而其讀寫速度又優(yōu)于磁帶。因此,將光盤作為分級存儲系統(tǒng)中的近線或離線存儲設(shè)備具有現(xiàn)實意義。本文在分布式文件系統(tǒng)基礎(chǔ)上,引入藍(lán)光光盤庫作為近線存儲設(shè)備,建立一個“在線——近線”分級存儲的云存儲平臺。在平臺設(shè)計和實現(xiàn)的過程中主要工作包括如下內(nèi)容:(1)對當(dāng)前的數(shù)據(jù)區(qū)分技術(shù)進(jìn)行研究與分析,在此基礎(chǔ)上,通過對用戶行為日志的分析,提取用戶行為關(guān)鍵特征,建立數(shù)據(jù)價值模型。該模型采用指數(shù)平滑法充分考慮各個時刻的數(shù)據(jù)價值,并根據(jù)數(shù)據(jù)價值的高低區(qū)分冷熱數(shù)據(jù)集,該模型能有效地提高云平臺中用戶訪問數(shù)據(jù)時命中在線存儲設(shè)備的概率。(2)研究數(shù)據(jù)遷移過程控制策略,實現(xiàn)數(shù)據(jù)在磁盤陣列和藍(lán)光存儲兩種不同存儲層次的設(shè)備之前自動的、不間斷的遷移。重點分析基于藍(lán)光光盤庫的數(shù)據(jù)遷移過程,有效地保證數(shù)據(jù)在遷移過程中的一致性。(3)采用元數(shù)據(jù)服務(wù)器與應(yīng)用服務(wù)器分離,前臺頁面與后臺服務(wù)分離的分模塊方式,實現(xiàn)平臺各個功能,充分體現(xiàn)低耦合的軟件設(shè)計思想。并且進(jìn)行一系列測試,測試結(jié)果表明平臺實現(xiàn)的數(shù)據(jù)遷移具有有效性和準(zhǔn)確性,能夠?qū)崿F(xiàn)數(shù)據(jù)在藍(lán)光光盤庫與磁盤之間高效傳輸,保證數(shù)據(jù)的一致性,釋放磁盤空間,并提高數(shù)據(jù)訪問的速度。
[Abstract]:With the wide application of the Internet in daily life, the amount of data in the network has increased exponentially. The IT industry has entered the storage era from the computing era. At present, many data centers are faced with the problems of huge system energy consumption, low utilization of storage resources and rising storage costs. Data show that storage costs account for 30-70% of the total cost of the data center. In the daily data generation, the frequently used data (hot data) accounts for only 20% of the total amount of data, and the data (cold data) that is rarely used or rarely used (cold data) is as high as 80%. In this case, there is a broad research and application prospect for hierarchical storage based on the characteristics of data cooling and heating, to give full play to the advantages of various storage devices and to reduce the total storage cost. Compared with the current development of storage media technology, the unit capacity of CD-ROMs is much lower than that of disks, and the speed of reading and writing is better than that of magnetic tape. Therefore, it is of practical significance to use optical disk as near-line or off-line storage device in hierarchical storage system. Based on the distributed file system, this paper introduces the Blu-ray optical disk library as the near-line storage device, and establishes a "on-line -near-line" hierarchical storage cloud storage platform. In the process of platform design and implementation, the main contents are as follows: (1) Research and analysis of current data differentiation technology, on the basis of which, through the analysis of user behavior log, the key features of user behavior are extracted. Establish data value model. In this model, the exponential smoothing method is used to fully consider the data value at each time, and the cold and hot data sets are distinguished according to the value of the data. This model can effectively improve the probability of users hitting online storage devices when accessing data in cloud platform. (2) Research data migration process control strategy, realize data before disk array and Blu-ray storage two different storage levels of devices automatically, An uninterrupted migration. The data migration process based on Blu-ray optical disk library is analyzed in order to ensure the consistency of data migration effectively. (3) the metadata server is separated from the application server, and the foreground page is separated from the background service. Realize each function of the platform, fully embody the low coupling software design idea. And a series of tests are carried out. The test results show that the data migration implemented by the platform is effective and accurate. It can efficiently transfer the data between the Blu-ray disc library and the disk, ensure the consistency of the data, and release the disk space. And improve the speed of data access.
【學(xué)位授予單位】：電子科技大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2017
【分類號】：TP333

【參考文獻(xiàn)】

相關(guān)期刊論文 前10條

1 張晶;黃小鋒;;一種基于微服務(wù)的應(yīng)用框架[J];計算機系統(tǒng)應(yīng)用;2016年09期

2 栗輝;唐萌;陳豪;;基于用戶行為分析的網(wǎng)站結(jié)構(gòu)優(yōu)化研究綜述[J];計算機科學(xué);2016年S1期

3 劉建華;張智雄;;保存元數(shù)據(jù)的發(fā)展趨勢研究[J];圖書館雜志;2016年06期

4 朱明哲;劉婷;;藍(lán)光技術(shù)應(yīng)用于廣電網(wǎng)絡(luò)儲存系統(tǒng)的可行性[J];中國有線電視;2016年02期

5 杜琳琳;;海量數(shù)據(jù)離線存儲系統(tǒng)研究[J];中國檔案;2016年02期

6 張滬寅;屈乾松;胡瑞蕓;;基于JSON的數(shù)據(jù)交換模型[J];計算機工程與設(shè)計;2015年12期

7 陳桂龍;;我國大數(shù)據(jù)發(fā)展?fàn)顩r[J];中國建設(shè)信息化;2015年Z1期

8 羅圣美;張佳程;周揚;程舒晗;;TiStor:基于策略引擎的分級存儲系統(tǒng)[J];小型微型計算機系統(tǒng);2015年08期

9 劉勝久;李天瑞;珠杰;;Zipf定律與網(wǎng)絡(luò)信息計量學(xué)[J];中文信息學(xué)報;2015年04期

10 趙偉東;;電子檔案藍(lán)光存儲應(yīng)用探究[J];檔案學(xué)研究;2015年03期

相關(guān)博士學(xué)位論文 前1條

1 鐘睿明;富云：一種跨越異構(gòu)云平臺的互備可靠云存儲系統(tǒng)的實現(xiàn)機制研究[D];北京郵電大學(xué);2014年

相關(guān)碩士學(xué)位論文 前7條

1 郭剛;內(nèi)存云分級混合存儲架構(gòu)數(shù)據(jù)遷移策略[D];新疆大學(xué);2016年

2 武琢;在線存儲系統(tǒng)的設(shè)計與實現(xiàn)[D];內(nèi)蒙古大學(xué);2015年

3 盧俊華;HDFS存儲機制的分析與研究[D];武漢理工大學(xué);2014年

4 劉曉然;基于文件的數(shù)據(jù)分級存儲的研究與實現(xiàn)[D];昆明理工大學(xué);2013年

5 王艷云;分級存儲系統(tǒng)中數(shù)據(jù)遷移技術(shù)的研究[D];西安建筑科技大學(xué);2012年

6 吳超龍;在線云存儲系統(tǒng)的設(shè)計與應(yīng)用[D];華南理工大學(xué);2010年

7 劉志寬;分級存儲系統(tǒng)中元數(shù)據(jù)管理的設(shè)計與實現(xiàn)[D];大連海事大學(xué);2008年

，

本文編號：2255343