發(fā)布時(shí)間：2018-10-11 12:52

【摘要】：近年來,隨著物聯(lián)網(wǎng)技術(shù)的應(yīng)用和“互聯(lián)網(wǎng)+”概念的提出,互聯(lián)網(wǎng)中的數(shù)據(jù)量呈出指數(shù)式的增長從而由信息時(shí)代進(jìn)入大數(shù)據(jù)時(shí)代。大數(shù)據(jù)時(shí)代的數(shù)據(jù)不僅在數(shù)據(jù)量上劇增還在結(jié)構(gòu)、分布、價(jià)值密度及維度等方面有所改變,這些數(shù)據(jù)在并行化快速處理上的要求需要具有半結(jié)構(gòu)化及非結(jié)構(gòu)化數(shù)據(jù)存儲(chǔ)、高擴(kuò)展性和高讀寫行能的非關(guān)系型數(shù)據(jù)庫及分布式計(jì)算模式來滿足。傳統(tǒng)關(guān)系型數(shù)據(jù)庫無法滿足處理大數(shù)據(jù)的要求,但是由于其廣泛應(yīng)用而在歷史上積累了大量的數(shù)據(jù),為了充分利用這些數(shù)據(jù)繼續(xù)為用戶提供服務(wù)、為企業(yè)創(chuàng)造更多的價(jià)值需要設(shè)計(jì)一種自動(dòng)化或半自動(dòng)化的遷移工具來完成傳統(tǒng)關(guān)系型數(shù)據(jù)庫向非關(guān)系型數(shù)據(jù)庫的轉(zhuǎn)換。另外,由于非關(guān)系型數(shù)據(jù)庫與傳統(tǒng)關(guān)系型數(shù)據(jù)庫的使用接口有著本質(zhì)上的區(qū)別,為了使遷移后原有業(yè)務(wù)應(yīng)用能夠適應(yīng)新的數(shù)據(jù)存儲(chǔ)模型推動(dòng)向非關(guān)系型數(shù)據(jù)庫的平滑過渡需要將非關(guān)系型數(shù)據(jù)庫的原生操作接口進(jìn)行封裝并通過設(shè)計(jì)一種SQL操作中間件來實(shí)現(xiàn)這個(gè)轉(zhuǎn)變。本文通過對(duì)數(shù)據(jù)庫存儲(chǔ)結(jié)構(gòu)、大數(shù)據(jù)相關(guān)技術(shù)和HBase的操作接口的研究,設(shè)計(jì)實(shí)現(xiàn)了數(shù)據(jù)庫遷移工具并對(duì)HBase的操作接口進(jìn)行了封裝。首先,對(duì)數(shù)據(jù)存儲(chǔ)和遷移的相關(guān)技術(shù)進(jìn)行了研究,深入地分析了傳統(tǒng)關(guān)系型數(shù)據(jù)庫的存儲(chǔ)原理和設(shè)計(jì)規(guī)則、HBase數(shù)據(jù)庫的架構(gòu)和存儲(chǔ)模式、Hadoop生態(tài)系統(tǒng)以及數(shù)據(jù)遷移設(shè)計(jì)的相關(guān)模型和技術(shù)。其次,針對(duì)傳統(tǒng)關(guān)系型數(shù)據(jù)庫的元數(shù)據(jù)進(jìn)行深入分析提出并設(shè)計(jì)實(shí)現(xiàn)了表模式和數(shù)據(jù)由MySQL向HBase遷移的算法。然后,對(duì)HBase的原生API進(jìn)行深入分析,利用SQL解析和Filter條件過濾并結(jié)合傳統(tǒng)關(guān)系型數(shù)據(jù)庫的操作習(xí)慣對(duì)其進(jìn)行了封裝。最后,通過設(shè)計(jì)實(shí)驗(yàn)進(jìn)行性能測(cè)試,分析結(jié)果表明本文所實(shí)現(xiàn)的遷移工具和操作中間件能夠快速有效地完成歷史數(shù)據(jù)的遷移工作并在以后的使用過程中滿足查詢需求,為數(shù)據(jù)存儲(chǔ)從關(guān)系型數(shù)據(jù)庫向非關(guān)系型數(shù)據(jù)庫的過渡提供了保障。
[Abstract]:In recent years, with the application of the Internet of things technology and the concept of "Internet", the amount of data in the Internet has increased exponentially, thus from the information age to big data era. The data of big data era not only increased sharply in data volume, but also changed in structure, distribution, value density and dimension, etc. These data need to be stored in semi-structured and unstructured data in parallelization and fast processing. Non-relational databases and distributed computing models with high scalability and high read-write performance are satisfied. The traditional relational database can not meet the requirement of processing big data, but because of its extensive application, it has accumulated a large amount of data in history. In order to make full use of this data, it continues to provide services to users. To create more value for enterprises, we need to design an automatic or semi-automatic migration tool to complete the transformation from traditional relational database to non-relational database. In addition, because the interface between non-relational database and traditional relational database is essentially different, In order to adapt the new data storage model to the smooth transition to non-relational database, the native operation interface of non-relational database is encapsulated and a SQL operation is designed. Middleware to achieve this transformation. Through the research of database storage structure, big data technology and HBase operation interface, this paper designs and implements the database migration tool and encapsulates the operation interface of HBase. Firstly, the related technologies of data storage and migration are studied. The storage principle and design rules of traditional relational database, the structure and storage mode of HBase database, the model and technology of Hadoop ecosystem and data migration design are analyzed. Secondly, an algorithm of table schema and data migration from MySQL to HBase is proposed and implemented by analyzing the metadata of traditional relational database. Then, the native API of HBase is deeply analyzed and encapsulated by using SQL parsing and Filter conditional filtering and combining with the operation habits of traditional relational database. Finally, through designing experiments to test the performance, the analysis results show that the migration tools and operation middleware implemented in this paper can quickly and effectively complete the migration of historical data and meet the query requirements in the later use process. It provides a guarantee for the transition of data storage from relational database to non-relational database.
【學(xué)位授予單位】：華僑大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2017
【分類號(hào)】：TP311.13

【參考文獻(xiàn)】

相關(guān)期刊論文 前10條

1 宋春紅;王佳斌;鄭力新;;一種MySQL到HBase的遷移策略的研究與實(shí)現(xiàn)[J];微型機(jī)與應(yīng)用;2016年13期

2 宋春紅;王佳斌;鄭力新;;結(jié)構(gòu)型數(shù)據(jù)庫向HBase遷移的方法對(duì)比分析[J];福建電腦;2016年05期

3 李青云;余文;;關(guān)系型數(shù)據(jù)庫到H Base的轉(zhuǎn)換設(shè)計(jì)[J];信息網(wǎng)絡(luò)安全;2015年01期

4 涂新莉;劉波;林偉偉;;大數(shù)據(jù)研究綜述[J];計(jì)算機(jī)應(yīng)用研究;2014年06期

5 李雪曉;葉云;田苗苗;黃劉生;;基于格的大數(shù)據(jù)動(dòng)態(tài)存儲(chǔ)完整性驗(yàn)證方案[J];信息網(wǎng)絡(luò)安全;2014年04期

6 韋海清;;淺談Java通過JDBC連接Oracle數(shù)據(jù)庫技術(shù)[J];計(jì)算機(jī)光盤軟件與應(yīng)用;2014年07期

7 高源;;NoSQL非關(guān)系型數(shù)據(jù)庫的發(fā)展和應(yīng)用研究[J];計(jì)算機(jī)光盤軟件與應(yīng)用;2014年05期

8 邱勝海;高成沖;王云霞;陸云;王志亮;馬銀忠;;大數(shù)據(jù)時(shí)代非關(guān)系型數(shù)據(jù)庫教學(xué)與實(shí)驗(yàn)改革探索[J];電腦知識(shí)與技術(shù);2013年31期

9 唐長城;楊峰;代棟;孫明明;周學(xué)海;;一種基于HBase的數(shù)據(jù)持久性和可用性研究[J];計(jì)算機(jī)系統(tǒng)應(yīng)用;2013年10期

10 陶雪嬌;胡曉峰;劉洋;;大數(shù)據(jù)研究綜述[J];系統(tǒng)仿真學(xué)報(bào);2013年S1期

相關(guān)碩士學(xué)位論文 前4條

1 劉珂;基于Hadoop平臺(tái)的大數(shù)據(jù)遷移與查詢方法研究及應(yīng)用[D];武漢理工大學(xué);2014年

2 唐韜;基于HBase的SQL轉(zhuǎn)化引擎研究[D];電子科技大學(xué);2014年

3 黎明;昆明學(xué)院圖書館信息系統(tǒng)分析與設(shè)計(jì)[D];云南大學(xué);2013年

4 劉欣然;基于異構(gòu)表的數(shù)據(jù)遷移算法研究[D];哈爾濱工程大學(xué);2013年

，

本文編號(hào)：2264262