面向并行數(shù)字地形分析的DEM數(shù)據(jù)云存儲方法研究
發(fā)布時間：2018-10-20 08:45
【摘要】：人類對地觀測技術的發(fā)展使得大范圍高精度地形數(shù)字高程模型(DEM)數(shù)據(jù)的數(shù)據(jù)量呈爆炸式增長�；趩我挥嬎銠C的傳統(tǒng)數(shù)字地形分析方法由于存儲與計算性能有限,無法滿足地學研究和生產(chǎn)應用的需求。面向新型硬件架構的并行數(shù)字地形分析理論和方法的研究已經(jīng)成為地學界的研究熱點之一。然而,目前的研究著重于考慮地形分析算法的并行,很少涉及面向并行數(shù)字地形分析的海量DEM數(shù)據(jù)以及數(shù)字地形分析結果的管理問題,已有的DEM數(shù)據(jù)管理模式也無法滿足并行數(shù)字地形分析的需求。分布式計算平臺Hadoop被認為是海量數(shù)據(jù)處理和分析的利器。如何利用Hadoop來管理急劇增加的DEM數(shù)據(jù),以滿足并行數(shù)字地形分析的需求,是值得研究的課題,本文即圍繞此課題展開研究,主要內容包括以下三個方面： 1)面向并行數(shù)字地形分析的DEM數(shù)據(jù)組織方式 在深入分析并行數(shù)字地形分析的特點及其對DEM數(shù)據(jù)組織和管理需求的基礎上,詳細介紹了DEM金字塔結構及其構建方法和數(shù)學模型,并基于此結構設計了DEM及其增量數(shù)據(jù)的分布式存儲模型,可實現(xiàn)海量DEM數(shù)據(jù)及其并行數(shù)字地形分析結果的有效存儲。 21面向并行數(shù)字地形分析的DEM數(shù)據(jù)管理方法 DEM管理方法包括空間索引機制、數(shù)據(jù)壓縮方法、系統(tǒng)容錯機制和并發(fā)訪問策略等。本文結合HBase數(shù)據(jù)庫的存儲模式,構建了三級空間索引,并提出了基于內容的空間索引方法,使系統(tǒng)同時支持基于特征查詢和基于內容查詢。在數(shù)據(jù)壓縮方面,設計了高程增量游程編碼壓縮算法,并對算法的適用性進行了實驗驗證。此外,深入研究了Hadoop的容錯方案和高并發(fā)訪問策略,并結合DEM數(shù)據(jù)的特點,給出了配置參數(shù)的建議值以及部分實現(xiàn)代碼。 3)DEM云存儲原型系統(tǒng)的設計與實現(xiàn) 通過分析DEM云存儲系統(tǒng)的設計需求,給出了系統(tǒng)的詳細設計方案。并以三臺PC機搭建Hadoop集群模擬DEM云存儲環(huán)境,在該集群環(huán)境上開發(fā)了DEM云存儲原型系統(tǒng)。該系統(tǒng)支持四種查詢方式：①基于文件名查詢；②基于范圍和分辨率查詢；③基于范圍和格網(wǎng)數(shù)查詢；④基于內容查詢。以全球SRTM3數(shù)據(jù)為實驗數(shù)據(jù)對系統(tǒng)進行驗證。實驗結果表明,數(shù)據(jù)檢索的結果是完全正確的,在現(xiàn)有的集群環(huán)境條件下,數(shù)據(jù)檢索的時間效率是令人滿意的。
[Abstract]:With the development of human earth observation technology, the data volume of (DEM) data of large range and high precision terrain digital elevation model increases explosively. The traditional digital terrain analysis method based on a single computer is unable to meet the requirements of geoscience research and production applications because of its limited storage and computing performance. The research of parallel digital terrain analysis theory and method for new hardware architecture has become one of the research hotspots in geoscience. However, the current research focuses on considering the parallelism of terrain analysis algorithms, and rarely involves the massive DEM data for parallel digital terrain analysis and the management of digital terrain analysis results. The existing DEM data management model can not meet the needs of parallel digital terrain analysis. Hadoop, a distributed computing platform, is considered to be a powerful tool for mass data processing and analysis. How to use Hadoop to manage the rapidly increasing DEM data to meet the needs of parallel digital terrain analysis is a topic worthy of study. The main contents include the following three aspects: 1) the DEM data organization for parallel digital terrain analysis is based on the in-depth analysis of the characteristics of parallel digital terrain analysis and the requirements for DEM data organization and management. This paper introduces the pyramid structure of DEM, its construction method and mathematical model in detail, and designs the distributed storage model of DEM and its incremental data based on this structure. The efficient storage of massive DEM data and its parallel digital terrain analysis results can be realized. 21 the DEM data management method for parallel digital terrain analysis includes spatial index mechanism, data compression method, DEM management method. System fault tolerant mechanism and concurrent access strategy. Combined with the storage mode of HBase database, this paper constructs a three-level spatial index, and proposes a content-based spatial index method, which enables the system to support both feature-based query and content-based query. In the aspect of data compression, the algorithm of height increment run-length coding compression is designed, and the applicability of the algorithm is verified by experiments. In addition, the fault-tolerant scheme and high concurrency access strategy of Hadoop are deeply studied, and the characteristics of DEM data are combined. The design and implementation of DEM cloud storage prototype system are given. By analyzing the design requirements of DEM cloud storage system, the detailed design scheme of the system is given. The Hadoop cluster simulation DEM cloud storage environment is built with three PC computers, and the DEM cloud storage prototype system is developed in the cluster environment. The system supports four query modes: (1) file name query; (2) range and resolution based query; (3) range and mesh query; (4) content based query. The global SRTM3 data are used as experimental data to verify the system. The experimental results show that the results of data retrieval are correct and the time efficiency of data retrieval is satisfactory under the existing cluster environment.
【學位授予單位】：南京師范大學
【學位級別】：碩士
【學位授予年份】：2013
【分類號】：P208
本文編號：2282643
[Abstract]:With the development of human earth observation technology, the data volume of (DEM) data of large range and high precision terrain digital elevation model increases explosively. The traditional digital terrain analysis method based on a single computer is unable to meet the requirements of geoscience research and production applications because of its limited storage and computing performance. The research of parallel digital terrain analysis theory and method for new hardware architecture has become one of the research hotspots in geoscience. However, the current research focuses on considering the parallelism of terrain analysis algorithms, and rarely involves the massive DEM data for parallel digital terrain analysis and the management of digital terrain analysis results. The existing DEM data management model can not meet the needs of parallel digital terrain analysis. Hadoop, a distributed computing platform, is considered to be a powerful tool for mass data processing and analysis. How to use Hadoop to manage the rapidly increasing DEM data to meet the needs of parallel digital terrain analysis is a topic worthy of study. The main contents include the following three aspects: 1) the DEM data organization for parallel digital terrain analysis is based on the in-depth analysis of the characteristics of parallel digital terrain analysis and the requirements for DEM data organization and management. This paper introduces the pyramid structure of DEM, its construction method and mathematical model in detail, and designs the distributed storage model of DEM and its incremental data based on this structure. The efficient storage of massive DEM data and its parallel digital terrain analysis results can be realized. 21 the DEM data management method for parallel digital terrain analysis includes spatial index mechanism, data compression method, DEM management method. System fault tolerant mechanism and concurrent access strategy. Combined with the storage mode of HBase database, this paper constructs a three-level spatial index, and proposes a content-based spatial index method, which enables the system to support both feature-based query and content-based query. In the aspect of data compression, the algorithm of height increment run-length coding compression is designed, and the applicability of the algorithm is verified by experiments. In addition, the fault-tolerant scheme and high concurrency access strategy of Hadoop are deeply studied, and the characteristics of DEM data are combined. The design and implementation of DEM cloud storage prototype system are given. By analyzing the design requirements of DEM cloud storage system, the detailed design scheme of the system is given. The Hadoop cluster simulation DEM cloud storage environment is built with three PC computers, and the DEM cloud storage prototype system is developed in the cluster environment. The system supports four query modes: (1) file name query; (2) range and resolution based query; (3) range and mesh query; (4) content based query. The global SRTM3 data are used as experimental data to verify the system. The experimental results show that the results of data retrieval are correct and the time efficiency of data retrieval is satisfactory under the existing cluster environment.
【學位授予單位】：南京師范大學
【學位級別】：碩士
【學位授予年份】：2013
【分類號】：P208
【參考文獻】
相關期刊論文 前6條
1 覃雄派;王會舉;杜小勇;王珊;;大數(shù)據(jù)分析——RDBMS與MapReduce的競爭與共生[J];軟件學報;2012年01期
2 方濤,李德仁,龔健雅,皮明紅;GeoImageDB多分辨率無縫影像數(shù)據(jù)庫系統(tǒng)的開發(fā)與實現(xiàn)[J];武漢測繪科技大學學報;1999年03期
3 陳仁喜;趙忠明;王殷行;;基于整型小波變換的DEM數(shù)據(jù)壓縮[J];武漢大學學報(信息科學版);2006年04期
4 多雪松;張晶;高強;;基于Hadoop的海量數(shù)據(jù)管理系統(tǒng)[J];微計算機信息;2010年13期
5 鄭晶晶;方金云;韓承德;;一種快速的DEM數(shù)據(jù)無損壓縮算法[J];系統(tǒng)仿真學報;2010年10期
6 趙立成,王素娟,施進明;國家衛(wèi)星氣象中心信息共享體制研究與技術實現(xiàn)[J];應用氣象學報;2002年05期
，本文編號：2282643
本文鏈接：http://sikaile.net/kejilunwen/dizhicehuilunwen/2282643.html
最近更新
教材專著
