本文選題：分布式數(shù)據(jù)去重復　切入點：差分壓縮　出處：《中國科學技術大學》2013年博士論文　論文類型：學位論文

【摘要】：隨著下一代廣播電視網(wǎng)(the Next Generation Broadcasting network, NGB)建設進程的推進和三網(wǎng)融合業(yè)務的開展,開放業(yè)務平臺上多樣化的雙向交互應用將廣播電視網(wǎng)的終端用戶從單一的數(shù)據(jù)被動接收者轉變?yōu)閿?shù)據(jù)生產(chǎn)者和消費者的雙重身份,這就向NGB提出了數(shù)據(jù)存儲需求。 基于用戶對數(shù)據(jù)方便、快捷、可靠共享方式的強烈需求,以及大部分廣播電視網(wǎng)智能終端自身存儲、計算資源有限的狀況,采用高可擴展、按需分配的集中式存儲資源池乃是符合信息產(chǎn)業(yè)技術發(fā)展趨勢的低成本、高效率解決方案。在這一需求背景下,在NGB中建設適合自身網(wǎng)絡特性的存儲服務具有重要意義。 另一方面,廣播電視網(wǎng)數(shù)字化整體轉換過程中形成的有服務器質(zhì)量保證的高清視頻類業(yè)務是廣播電視網(wǎng)區(qū)別于互聯(lián)網(wǎng)和寬帶電信網(wǎng)的特色業(yè)務。與其他數(shù)據(jù)的共享需求不同,高清視頻主要通過直播、點播和錄播的方式欣賞,錄制的大容量視頻文件保存在高清媒體終端本地的存儲設備中。由于存儲設備普遍使用價格可接受、但訪問性能提升逐漸趨于平緩的傳統(tǒng)硬盤,不足以應付視頻分辨率的幾何式增長速度；加快高清視頻錄制播放時硬盤文件系統(tǒng)的并行讀寫速度,降低文件讀寫時處理器的使用率,保證用戶欣賞時的流暢體驗就顯得尤為重要。 針對這兩方面的問題,本文的主要工作和創(chuàng)新點如下： 1.利用廣播電視網(wǎng)絡下行帶寬大、且可管理分配的優(yōu)勢,提出一種參考差分壓縮的去重復傳輸優(yōu)化策略。在該傳輸策略中,存儲服務器根據(jù)客戶端發(fā)送的分塊摘要值在數(shù)據(jù)庫中查找相似文件和相似分塊,并將相似分塊回復給客戶端,客戶端將相似分塊與本地數(shù)據(jù)進行差分壓縮編碼后上傳更少量的數(shù)據(jù)。通過將一部分數(shù)據(jù)量從低帶寬的一端移到高帶寬的一端,而又不顯著增加總的傳輸量,參考差分的去重傳輸策略減少了數(shù)據(jù)傳輸?shù)目倳r間,加快了數(shù)據(jù)同步的速度。 2.提出一種Hash表與B+樹相結合的目錄索引結構HB-Index。使用Hash表索引的方式,可以在O(1)時間內(nèi)找到文件名對應的索引節(jié)點；另外僅在目錄文件中保存子文件的索引節(jié)點號,并由B+樹組織這些子索引節(jié)點號,支持O(logdN)時間內(nèi)文件的創(chuàng)建和刪除。實測結果表明,HB-Index不僅能夠快速地處理包含大量文件的單個文件夾,在分區(qū)內(nèi)文件總數(shù)很大時,也有優(yōu)越的全局性能。 3.提出一種磁盤文件系統(tǒng)-Boaz,用于提升高清視頻的快速連續(xù)讀寫速度。它將元數(shù)據(jù)與數(shù)據(jù)分區(qū)域存儲,設置合適的最小分配單元,在保證大文件最低連續(xù)讀寫帶寬和最大磁盤空間利用率的同時,降低了寫文件時的元數(shù)據(jù)更新頻率和CPU資源消耗。另外,基于預期文件大小的數(shù)據(jù)塊預分配策略增加了文件元數(shù)據(jù)的連續(xù)性,降低磁盤的老化速度。在高清視頻播放器上的測試結果表明,Boaz在順序讀寫和并行訪問大文件時,能夠獲得比現(xiàn)有磁盤文件系統(tǒng)更快的訪問速度和更低的CPU利用率。 本文針對幾個關鍵問題提出的優(yōu)化解決方法,或許可以為NGB中終端用戶數(shù)據(jù)的存儲研究起到磚引玉的作用,在進一步完善之后能夠為NGB的示范建設進程提供一些實際可應用的技術。
[Abstract]:With the next generation of broadcast television networks (the Next Generation Broadcasting network, NGB) and the advance of the construction of triple play business, open service platform interactive application diversification will end users of radio and television networks data from passive receivers into a single double identity data producers and consumers, it is proposed to NGB the data storage needs.
Based on data for user convenience, fast, strong demand for reliable sharing, as well as most radio and television network intelligent terminal storage, limited computing resources, a highly scalable, centralized storage resource pool according to needs is the low cost development trend of information technology industry, high efficiency in this demand solutions. Under the background, the construction of storage services for their own network characteristics has important significance in NGB.
On the other hand, the formation of the overall conversion of digital broadcast television network in the process of HD video service server to ensure the quality of radio and television network from the Internet and broadband telecommunication network business characteristics. Different from other data sharing requirements, mainly through direct broadcast high-definition video on demand, appreciation and taped the way large video files the record stored in the HD media storage device terminal in the local storage device. Due to the widespread use of the price is acceptable, but the access performance gradually flatten the traditional hard disk, is not enough to cope with geometric video resolution growth rate; speed up the parallel read and write speed HD video recording playback when the hard disk file system, reduce the use of documents to read write processor rate, to ensure that users enjoy the smooth experience is particularly important.
In view of these two aspects, the main work and innovation of this paper are as follows:
1. use of radio and television network downlink bandwidth, and management advantages, the paper proposes a reference differential compression to repeat transmission optimization strategy. In this scheme, the storage server according to block the client value in the database to find similar documents and similar blocks, and similar block reply the client, the client will be similar to the block and the local data are differential data upload less compression encoding. Through a portion of the amount of data from one end of the move to the end of the low bandwidth high bandwidth, but not significantly increase the total amount of transmissions, reference differential transmission strategy to reduce the total time of data transmission and the speed of data synchronization.
HB-Index. directory structure 2. presents a Hash table and B+ tree combined with Hash index method, O (1) found in the index node file name corresponding to the time; save the inode number sub file only in the directory file, and the B+ tree to organize these sub index nodes. Support O (logdN) to create and delete files in time. The measured results show that the single folder HB-Index can not only quickly deal with large numbers of files, the total partition file is very large, there are overall superior performance.
3. presents a disk file system -Boaz, to enhance the high-definition video rapid sequential read and write speed. It will metadata and data storage area, set the minimum allocation unit is appropriate, in which the minimum file sequential read and write bandwidth and maximum disk space utilization and reduce the metadata update frequency and write files CPU resource consumption. In addition, the expected file size of the data block allocation strategy based on increasing the continuity of file metadata, reduce the aging speed of the disk. The test results in the HD video player on the show, Boaz read and write access to large files in parallel order, can be obtained than the existing disk file system faster access and the lower CPU utilization rate.
The optimization solution proposed for several key problems in this paper may play a role of brick and jade in the research of end user data storage in NGB. After further improvement, it can provide some practical and applied technologies for NGB's demonstration and construction process.

【學位授予單位】：中國科學技術大學
【學位級別】：博士
【學位授予年份】：2013
【分類號】：TN946.3;TP333.35

【相似文獻】

相關期刊論文 前10條

1 易法令;謝長生;吳非;;體全息數(shù)據(jù)存儲文件系統(tǒng)空間分配策略研究[J];計算機應用;2006年08期

2 潘燕梅;;基于Nucleus的閃存文件系統(tǒng)研究和改進[J];傳奇.傳記文學選刊(理論研究);2011年01期

3 紀兆輝;胡孔法;;基于ADO.Net2.0實現(xiàn)數(shù)據(jù)庫系統(tǒng)中BLOB數(shù)據(jù)的處理[J];淮海工學院學報(自然科學版);2009年02期

4 于曉;王欣;閻豐;隋永新;金春水;楊懷江;;電暈探測系統(tǒng)中JPEG截圖文件系統(tǒng)的設計[J];光學精密工程;2005年06期

5 劉海;朱紅;;基于LPC2148的SD卡實現(xiàn)嵌入式系統(tǒng)升級設計[J];國外電子元器件;2008年10期

6 仝得一;肖連華;馮麗芬;;757計算機文件系統(tǒng)的實現(xiàn)[J];計算機研究與發(fā)展;1985年01期

7 王懷毅;;西文編目——名稱規(guī)范文件系統(tǒng)的建立[J];中國圖書館學報;1987年01期

8 洪U喿，

本文編號：1610554