【摘要】：隨著互聯(lián)網(wǎng)的快速發(fā)展,應用所產(chǎn)生的數(shù)據(jù)量越來越大。分布式數(shù)據(jù)庫HBase對海量數(shù)據(jù)的管理得到了廣泛應用。許多企業(yè)希望將原本存放在關系型數(shù)據(jù)庫中的數(shù)據(jù)遷移到分布式數(shù)據(jù)庫HBase中,并在HBase上對數(shù)據(jù)進行管理。因此研究在HBase上提供一個數(shù)據(jù)管理系統(tǒng)具有重要意義。在分析基于HBase的數(shù)據(jù)管理系統(tǒng)設計目標基礎上,給出了系統(tǒng)的整體設計方案,包括兩大功能:將關系型數(shù)據(jù)庫中的模式和數(shù)據(jù)遷移到HBase中,使用SQL語句管理HBase中的數(shù)據(jù)。關系型數(shù)據(jù)庫模式和數(shù)據(jù)遷移功能將關系型數(shù)據(jù)庫表的列信息、索引信息、主外鍵信息存儲到HBase的元數(shù)據(jù)表中。表數(shù)據(jù)遷移任務分割為多個小任務并盡可能地平分給集群中機器來執(zhí)行,同時根據(jù)主外鍵信息對數(shù)據(jù)進行冗余存儲,根據(jù)索引信息在HBase中創(chuàng)建索引表并記錄索引數(shù)據(jù)。使用SQL語句管理HBase中數(shù)據(jù),重點優(yōu)化多表連接查詢。將多表連接查詢?nèi)蝿崭鶕?jù)HBase的特點分解為若干子多表連接查詢,使用HBase的協(xié)處理器并發(fā)的執(zhí)行子多表連接查詢。子多表連接查詢中根據(jù)連接條件的特點優(yōu)化表的連接順序,利用遷移流程產(chǎn)生的冗余數(shù)據(jù)、索引數(shù)據(jù)提高連接查詢效率。子多表連接查詢的中間數(shù)據(jù)利用哈希表和多叉樹存儲來降低內(nèi)存開銷。在客戶端合并多個子多表連接查詢返回的結(jié)果。通過對基于HBase數(shù)據(jù)管理系統(tǒng)的實驗測試,表明該系統(tǒng)能高效的遷移表模式和數(shù)據(jù),對遷移后的數(shù)據(jù)能正確的管理,且在多表連接查詢時相比與Hive具有較好性能。
[Abstract]:With the rapid development of the Internet, the amount of data generated by the application is increasing. Distributed database HBase has been widely used in the management of massive data. Many enterprises want to migrate the data stored in the relational database to the distributed database HBase and manage the data on the HBase. Therefore, it is of great significance to provide a data management system on HBase. Based on the analysis of the design objectives of the data management system based on HBase, the overall design scheme of the system is presented, which includes two functions: migrating the schema and data from the relational database to the HBase, and managing the data in the HBase by using the SQL statement. Relational database schema and data migration function store the column information, index information and primary foreign key information of relational database table in HBase metadata table. The task of table data migration is divided into several small tasks and distributed equally to the machines in the cluster as far as possible. At the same time, the data is stored redundant according to the primary foreign key information, and the index table is created in HBase according to the index information and the index data is recorded. Use SQL statements to manage data in HBase, focusing on optimizing multi-table join queries. According to the characteristics of HBase, the task of multi-table join query is decomposed into several sub-multi-table join queries, and the co-processor of HBase is used to execute the sub-multi-table join query. In order to optimize the join order of subtable join query according to the characteristics of join condition, the redundant data generated by migration process is used to improve the efficiency of join query. The intermediate data of sub-table join query uses hash table and multi-tree storage to reduce memory overhead. Results returned by merging multiple child multiple table join queries on the client side. The experimental results based on HBase data management system show that the system can efficiently migrate the table pattern and data, manage the migrated data correctly, and have better performance than Hive in multi-table join query.
【學位授予單位】：華中科技大學
【學位級別】：碩士
【學位授予年份】：2016
【分類號】：TP311.13

【相似文獻】

相關期刊論文 前10條

1 楊武,文守遜;匯總型多表連接查詢的一種優(yōu)化方法[J];計算機系統(tǒng)應用;2000年01期

2 徐帆;匯總型多表連接查詢的一種優(yōu)化方法[J];計算機工程與設計;2002年10期

3 張雷;唐桂芬;蘇冉冉;;基于通用空間連接圖的適應性多元空間連接查詢[J];計算機光盤軟件與應用;2013年13期

4 彭建平,王變琴;再探多連接查詢優(yōu)化方法[J];中山大學學報(自然科學版);2001年02期

5 劉宇,孫莉,田永青;并行空間連接查詢處理[J];上海交通大學學報;2002年04期

6 王果,徐仁佐;結(jié)合哈希過濾的一種改進多連接查詢優(yōu)化算法[J];計算機工程;2004年07期

7 陳恕勝;劉衛(wèi)東;;基于圖的適應性多連接查詢優(yōu)化算法[J];計算機工程;2009年10期

8 郭聰莉;朱莉;李向;;基于蟻群算法的多連接查詢優(yōu)化方法[J];計算機工程;2009年10期

9 王，

本文編號：2275687