發(fā)布時(shí)間：2018-01-15 00:27

  本文關(guān)鍵詞：非結(jié)構(gòu)化數(shù)據(jù)統(tǒng)一存儲(chǔ)平臺(tái)的設(shè)計(jì)與實(shí)現(xiàn)　出處：《浙江大學(xué)》2013年碩士論文　論文類型：學(xué)位論文

  更多相關(guān)文章： 非結(jié)構(gòu)化數(shù)據(jù) 統(tǒng)一存儲(chǔ) 批處理

【摘要】：當(dāng)今互聯(lián)網(wǎng)上的數(shù)據(jù)正在呈現(xiàn)出迅速增長(zhǎng)的發(fā)展趨勢(shì),這種趨勢(shì)不僅僅體現(xiàn)在數(shù)據(jù)的數(shù)量上,同時(shí)也體現(xiàn)在數(shù)據(jù)的種類上。從傳統(tǒng)的文本數(shù)據(jù)到如今的網(wǎng)絡(luò)文檔、圖片、音頻以及視頻,互聯(lián)網(wǎng)數(shù)據(jù)的主流逐漸從結(jié)構(gòu)化數(shù)據(jù)轉(zhuǎn)變?yōu)榉墙Y(jié)構(gòu)數(shù)據(jù),而這些日益增長(zhǎng)并種類繁多的非結(jié)構(gòu)化數(shù)據(jù),為互聯(lián)網(wǎng)數(shù)據(jù)的存儲(chǔ)管理帶來(lái)了新的挑戰(zhàn)。 本文首先研究了針對(duì)各類海量非結(jié)構(gòu)化數(shù)據(jù)的存儲(chǔ)問(wèn)題所提出的解決方案,分析出各存儲(chǔ)系統(tǒng)所存在的問(wèn)題,從而總結(jié)出實(shí)現(xiàn)非結(jié)構(gòu)化數(shù)據(jù)統(tǒng)一存儲(chǔ)的關(guān)鍵問(wèn)題。 然后,針對(duì)具有海量、異構(gòu)、關(guān)聯(lián)等特征的非結(jié)構(gòu)化數(shù)據(jù)的存儲(chǔ)問(wèn)題,提出了非結(jié)構(gòu)化數(shù)據(jù)統(tǒng)一存儲(chǔ)管理平臺(tái)D-Ocean Repository,通過(guò)解決元數(shù)據(jù)管理、統(tǒng)一數(shù)據(jù)接口、異構(gòu)存儲(chǔ)以及數(shù)據(jù)的高可用性與一致性等關(guān)鍵問(wèn)題,融合了HDFS, HBase, MySQL, XMLDB等各類存儲(chǔ)設(shè)施,并通過(guò)異構(gòu)存儲(chǔ)設(shè)施的選擇機(jī)制,解決各類數(shù)據(jù)的高效混合存儲(chǔ)問(wèn)題。 同時(shí),基于統(tǒng)一存儲(chǔ)平臺(tái),本文設(shè)計(jì)并實(shí)現(xiàn)了一個(gè)非結(jié)構(gòu)數(shù)據(jù)的批處理框架,利用數(shù)據(jù)統(tǒng)一存儲(chǔ)的特性,解決了各類非結(jié)構(gòu)化數(shù)據(jù)的統(tǒng)一處理問(wèn)題,并基于MapReduce架構(gòu)實(shí)現(xiàn)了數(shù)據(jù)的高效并行處理,使得計(jì)算資源與數(shù)據(jù)存儲(chǔ)得到有機(jī)結(jié)合。 最后,本文還實(shí)現(xiàn)了一個(gè)使用D-Ocean系統(tǒng)作為后臺(tái)數(shù)據(jù)管理的互聯(lián)網(wǎng)應(yīng)用——互聯(lián)網(wǎng)跨媒體新聞檢索系統(tǒng),用以證明非結(jié)構(gòu)化數(shù)據(jù)統(tǒng)一存儲(chǔ)平臺(tái)的實(shí)用性,有助于未來(lái)面向更多非結(jié)構(gòu)化數(shù)據(jù)的互聯(lián)網(wǎng)應(yīng)用實(shí)現(xiàn)。
[Abstract]:The data on the Internet is showing a trend of rapid growth, this trend is not only reflected in the amount of data, but also reflected in the types of data. Images from the traditional text data to web documents, audio and video, mainstream Internet data gradually transformed from structured data for unstructured data however, the increasing and many kinds of unstructured data, brings new challenges to the Internet data storage management.
In this paper, we first study the solutions proposed for the storage problem of all kinds of massive unstructured data, analyze the problems existing in each storage system, and summarize the key problems of unified storage of unstructured data.
Then, for a massive, heterogeneous, unstructured data storage problems associated with such features, the unstructured data storage management platform D-Ocean Repository, the solution of metadata management, unified data interface, high availability and consistency of the key issues of heterogeneous storage and data fusion, HDFS, HBase, MySQL, XMLDB other types of storage facilities, and through the selection mechanism of heterogeneous storage facilities, solve the problems of various efficient hybrid storage data.
At the same time, based on the unified storage platform, this paper designs and implements a non structured data batch processing framework, using the characteristics of data storage, to solve the problem of unified processing of unstructured data, and based on the MapReduce architecture to realize efficient parallel processing of data, making the computing resources and data storage are combined.
Finally, this paper implements a D-Ocean system as the background data management applications of the Internet - Internet media retrieval system, used to prove the viability of unstructured data storage platform, help for the future application of the realization of the Internet more non structured data.

【學(xué)位授予單位】：浙江大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2013
【分類號(hào)】：TP333

【參考文獻(xiàn)】

相關(guān)期刊論文 前1條

1 李慧,顏顯森;數(shù)據(jù)庫(kù)技術(shù)發(fā)展的新方向——非結(jié)構(gòu)化數(shù)據(jù)庫(kù)[J];情報(bào)理論與實(shí)踐;2001年04期

，

本文編號(hào)：1426007