【摘要】：地質(zhì)數(shù)據(jù)采集方式的多樣性導致了數(shù)據(jù)規(guī)模的不斷增長,已經(jīng)達到了“地質(zhì)大數(shù)據(jù)”的5“V”特點,數(shù)據(jù)管理和分析處理的復雜程度不斷增加,使得對海量地質(zhì)數(shù)據(jù)進行高效運維和數(shù)據(jù)挖掘的難度不斷增大,迫切需要新的技術(shù)手段來實現(xiàn)地質(zhì)數(shù)據(jù)的智能化服務(wù)和挖掘地質(zhì)數(shù)據(jù)的潛在價值。分布式存儲和云計算提供了一種解決上述問題的新思路。Hadoop大數(shù)據(jù)處理技術(shù)得到了國內(nèi)外研究者們越多越多的關(guān)注,成為海量數(shù)據(jù)存儲、計算、挖掘技術(shù)的研究熱點。本文旨在基于搭建虛擬化地質(zhì)云平臺,實現(xiàn)積累的地質(zhì)數(shù)據(jù)能夠共享和互操作。深入研究和探索Hadoop集群中的HDFS分布式文件系統(tǒng)、Map Reduce并行編程框架、Hbase列式存儲數(shù)據(jù)庫等組件,結(jié)合全國地質(zhì)礦產(chǎn)潛力評價數(shù)據(jù),將Hadoop技術(shù)應(yīng)用于地質(zhì)大數(shù)據(jù)分析研究中。本文的主要工作如下:(1)通過對云計算和大數(shù)據(jù)的研究,闡述了其概念、關(guān)鍵技術(shù)等內(nèi)容并提出了地質(zhì)云平臺的體系結(jié)構(gòu),重點闡述了開源云計算和存儲框架Hadoop,尤其是分布式文件系統(tǒng)HDFS、并行計算框架Map Reduce和列式存儲Hbase。(2)通過對海量地質(zhì)數(shù)據(jù)整合、共享和查詢檢索的需求分析,利用分布式存儲技術(shù)和虛擬化技術(shù)設(shè)計、搭建了Master/Slave架構(gòu)的云數(shù)據(jù)計算與存儲集群平臺。利用Hadoop系統(tǒng)中的HDFS和Map Reduce,為我們設(shè)計海量地質(zhì)數(shù)據(jù)存儲架構(gòu)提供了有力的技術(shù)支撐,最終實現(xiàn)在高并發(fā)、高負載的集群環(huán)境中對地質(zhì)數(shù)據(jù)進行高效訪問。(3)從Hadoop集群的云存儲出發(fā),解決了小文件在HDFS里合并存儲的優(yōu)化,使用Map Reduce算法使合并過程效率更高。同時通過整體考慮各個負載因素,采用信息熵算法確定權(quán)重值,經(jīng)過多輪負載均衡,提高系統(tǒng)應(yīng)對高并發(fā)情況,優(yōu)化文件讀寫,系統(tǒng)效率有了極大提高。(4)研究了架構(gòu)在虛擬云平臺上的HBase數(shù)據(jù)庫,根據(jù)礦產(chǎn)潛力評價數(shù)據(jù)的表特點設(shè)計rowkey,提高了地質(zhì)大數(shù)據(jù)存儲管理、查詢檢索的效率。通過與O racle關(guān)系數(shù)據(jù)庫的數(shù)據(jù)入庫、數(shù)據(jù)檢索對比實驗,驗證了HBase在處理海量地質(zhì)數(shù)據(jù)方面的優(yōu)越性。
[Abstract]:The diversity of geological data collection methods has led to the continuous growth of data scale, which has reached the 5 "V" characteristic of "geological big data", and the complexity of data management and analysis has been increasing. It is becoming more and more difficult to carry out efficient transportation and data mining for massive geological data. Therefore, new technical means are urgently needed to realize the intelligent service of geological data and the potential value of mining geological data. Distributed storage and cloud computing provide a new way to solve the above problems. Hadoop big data technology has attracted more and more attention from researchers at home and abroad, and has become a research hotspot in mass data storage, computing and mining technology. The purpose of this paper is to share and interoperate the accumulated geological data based on the virtual geological cloud platform. In this paper, we deeply study and explore the components of HDFS distributed file system (HDFS) in Hadoop cluster, such as HDFS Reduce parallel programming framework, Hbase column storage database and so on. Combined with the evaluation data of geological and mineral potential in China, Hadoop technology is applied to the analysis and research of geological big data. The main work of this paper is as follows: (1) through the research of cloud computing and big data, the concept and key technology of cloud computing are expounded, and the architecture of geological cloud platform is put forward. The requirements of open source cloud computing and storage framework (Hadoop,), especially distributed file system (HDFS,) parallel computing framework (Map Reduce) and column storage Hbase. (2), are analyzed by integrating, sharing and querying massive geological data. A cloud data computing and storage cluster platform based on Master/Slave architecture is built by using distributed storage technology and virtualization technology. The use of HDFS and Map Reduce, in the Hadoop system provides a powerful technical support for us to design a massive geological data storage architecture, which is finally implemented in high concurrency. The geological data is accessed efficiently in the high-load cluster environment. (3) based on the cloud storage of Hadoop cluster, the optimization of merging and storing small files in HDFS is solved, and the Map Reduce algorithm is used to make the merging process more efficient. At the same time, considering all the load factors as a whole, using the information entropy algorithm to determine the weight value, after the multi-wheel load balancing, improve the system to deal with the high concurrency, optimize the file reading and writing, The system efficiency has been greatly improved. (4) the HBase database based on virtual cloud platform is studied. According to the table characteristics of mineral potential evaluation data, rowkey, is designed to improve the efficiency of geological big data storage management and query retrieval. The superiority of HBase in dealing with massive geological data is verified by the data input and data retrieval contrast experiment with O racle relational database.
【學位授予單位】：湖南科技大學
【學位級別】：碩士
【學位授予年份】：2017
【分類號】：P628

【參考文獻】

相關(guān)期刊論文 前10條

1 林文煜;戴青云;曹江中;何小明;李能;;一種基于內(nèi)容的海量圖像檢索框架的設(shè)計與實現(xiàn)[J];電腦知識與技術(shù);2016年09期

2 譚永杰;;地質(zhì)大數(shù)據(jù)與信息服務(wù)工程技術(shù)框架[J];地理信息世界;2016年01期

3 陳靜;;基于Hadoop云計算平臺的文本處理算法的研究與改進[J];天津科技;2016年01期

4 朱月琴;譚永杰;張建通;毛波;沈婕;汲超飛;;基于Hadoop的地質(zhì)大數(shù)據(jù)融合與挖掘技術(shù)框架[J];測繪學報;2015年S1期

5 邵奇峰;李楓;;一種基于HBase的空間關(guān)鍵字查詢算法[J];計算機工程與科學;2015年11期

6 李朝奎;嚴雯英;肖克炎;趙亞楠;;地質(zhì)大數(shù)據(jù)分析與應(yīng)用模式研究[J];地質(zhì)學刊;2015年03期

7 李超嶺;李豐丹;李健強;劉園園;劉暢;呂霞;;智能地質(zhì)調(diào)查體系與架構(gòu)[J];中國地質(zhì);2015年04期

8 嚴光生;薛群威;肖克炎;陳建平;繆謹勵;余海龍;;地質(zhì)調(diào)查大數(shù)據(jù)研究的主要問題分析[J];地質(zhì)通報;2015年07期

9 陳建平;李婧;崔寧;于萍萍;;大數(shù)據(jù)背景下地質(zhì)云的構(gòu)建與應(yīng)用[J];地質(zhì)通報;2015年07期

10 趙鵬大;;大數(shù)據(jù)時代數(shù)字找礦與定量評價[J];地質(zhì)通報;2015年07期

相關(guān)博士學位論文 前2條

1 李源林;基于服務(wù)器虛擬化的網(wǎng)絡(luò)GIS集群關(guān)鍵技術(shù)研究[D];中國地質(zhì)大學;2013年

2 康俊鋒;云計算環(huán)境下高分辨率遙感影像存儲與高效管理技術(shù)研究[D];浙江大學;2011年

相關(guān)碩士學位論文 前7條

1 張振猛;基于Hadoop的海量文件存儲系統(tǒng)的分析與設(shè)計[D];北京工業(yè)大學;2015年

2 李潔;基于Hadoop的海量視頻的分布式存儲與檢索研究[D];南京郵電大學;2015年

3 張衛(wèi)東;基于Hadoop的海量圖片云存儲系統(tǒng)研究與設(shè)計[D];中國海洋大學;2014年

4 陳時遠;基于HDFS的分布式海量遙感影像數(shù)據(jù)存儲技術(shù)研究[D];中國科學院大學(工程管理與信息技術(shù)學院);2013年

5 張新榮;基于HBase的小文件存儲系統(tǒng)的研究及實現(xiàn)[D];東北大學;2012年

6 劉浩;基于負載均衡的存儲架構(gòu)研究與應(yīng)用[D];山東大學;2011年

7 孔舟;分布式虛擬化計算平臺高可靠任務(wù)拆分系統(tǒng)的設(shè)計與實現(xiàn)[D];電子科技大學;2011年

，

本文編號：2201822