本文選題：MapReduce + 柵格數(shù)據(jù)��； 參考：《江西理工大學(xué)》2013年碩士論文

【摘要】：隨著對地觀測技術(shù)的迅速發(fā)展，柵格數(shù)據(jù)量急劇增加，傳統(tǒng)單節(jié)點架構(gòu)的GIS系統(tǒng)已無法滿足大規(guī)模柵格數(shù)據(jù)處理和分析的需求，如何在分布式并行環(huán)境下提高大規(guī)模柵格數(shù)據(jù)空間分析算法的效率，已成為目前地學(xué)領(lǐng)域研究的重點。柵格數(shù)據(jù)空間分析具有數(shù)據(jù)量和計算量大的特點，屬于典型的數(shù)據(jù)密集型計算。目前，業(yè)界提出了多種并行計算模式，相對于傳統(tǒng)的MPI并行編程模型，開源Hadoop框架下的MapReduce并行編程模型更適用于數(shù)據(jù)密集型計算，同時具有較高的性能。因此本文將并行編程模型MapReduce和柵格數(shù)據(jù)空間分析典型算法相結(jié)合，主要解決大規(guī)模柵格數(shù)據(jù)計算效率低的問題。 本文從大規(guī)模柵格數(shù)據(jù)并行的角度，對數(shù)據(jù)劃分、數(shù)據(jù)并行導(dǎo)入和結(jié)果融合進行分析，并在此基礎(chǔ)上設(shè)計柵格數(shù)據(jù)空間分析并行化算法。主要做了下面工作：首先，針對大規(guī)模柵格數(shù)據(jù)的特點，提出了Hadoop框架下利用分布式文件系統(tǒng)HDFS構(gòu)建高效的數(shù)據(jù)組織模型，并針對柵格處理中鄰域型算法的數(shù)據(jù)邊界問題，提出了柵格數(shù)據(jù)重分塊處理機制；其次，針對傳統(tǒng)串行數(shù)據(jù)讀取速度慢的問題，設(shè)計基于MapRedcue的柵格金字塔并行構(gòu)建，實現(xiàn)大規(guī)模柵格數(shù)據(jù)的并行導(dǎo)入；然后，結(jié)合MapReduce并行編程模型，設(shè)計基本地形因子和地形特征提取的并行化算法，，以提高大規(guī)模柵格數(shù)據(jù)空間分析的效率；最后，與串行算法做了對比實驗，驗證了柵格數(shù)據(jù)空間分析并行化算法的效率。結(jié)果表明，基于MapReduce的柵格數(shù)據(jù)空間分析并行化算法效果較好。同時，隨著數(shù)據(jù)節(jié)點和數(shù)據(jù)量的增加，并行化算法的效率逐步提高。 因此，本文設(shè)計的基于MapReduce的柵格數(shù)據(jù)空間分析并行化算法有效提升了大規(guī)模柵格數(shù)據(jù)的計算效率。
[Abstract]:With the rapid development of Earth observation technology, the amount of grid data increases rapidly, and the traditional single-node GIS system can not meet the needs of large-scale grid data processing and analysis. How to improve the efficiency of large scale raster spatial analysis algorithm in distributed parallel environment has become the focus of geoscience research. Raster data space analysis has the characteristics of large amount of data and computation, so it is a typical data intensive calculation. At present, many parallel computing models have been proposed in the industry. Compared with the traditional MPI parallel programming model, the MapReduce parallel programming model based on open source Hadoop framework is more suitable for data-intensive computing and has higher performance. So the parallel programming model MapReduce and the typical algorithms of raster data space analysis are combined to solve the problem of low efficiency of large scale raster data computation. From the point of view of large scale raster data parallelism, this paper analyzes data partition, data parallel import and result fusion, and then designs a parallel algorithm for raster data space analysis. The main work is as follows: firstly, according to the characteristics of large-scale raster data, an efficient data organization model based on distributed file system (HDFS) under Hadoop framework is proposed, and the data boundary problem of neighborhood algorithm in grid processing is also discussed. Secondly, aiming at the problem of slow reading speed of traditional serial data, the parallel construction of grid pyramid based on MapRedcue is designed to realize the parallel import of large scale raster data. Combined with MapReduce parallel programming model, a parallel algorithm for extracting basic terrain factors and terrain features is designed to improve the efficiency of large scale raster data space analysis. The efficiency of parallel algorithm for raster data space analysis is verified. The results show that the parallel algorithm of raster data space analysis based on MapReduce is effective. At the same time, with the increase of data nodes and data, the efficiency of parallelization algorithm is improved gradually. Therefore, the parallel algorithm of raster data space analysis based on MapReduce in this paper can effectively improve the computational efficiency of large scale raster data.
【學(xué)位授予單位】：江西理工大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2013
【分類號】：P208

【參考文獻】

相關(guān)期刊論文 前10條

1 程承旗;張恩東;萬元嵬;宋樹華;;遙感影像剖分金字塔研究[J];地理與地理信息科學(xué);2010年01期

2 王結(jié)臣;王豹;胡瑋;張輝;;并行空間分析算法研究進展及評述[J];地理與地理信息科學(xué);2011年06期

3 李建江;崔健;王聃;嚴林;黃義雙;;MapReduce并行編程模型研究綜述[J];電子學(xué)報;2011年11期

4 程果;景寧;陳犖;熊偉;歐陽柳;;柵格數(shù)據(jù)處理中鄰域型算法的并行優(yōu)化方法[J];國防科技大學(xué)學(xué)報;2012年04期

5 江嶺;湯國安;劉凱;宋效東;陽建逸;張剛;;局部型地形因子并行計算方法研究[J];地球信息科學(xué)學(xué)報;2012年06期

6 宋效東;竇萬峰;湯國安;江嶺;趙菁;趙明偉;;分布式并行地形分析中數(shù)據(jù)劃分機制研究[J];國防科技大學(xué)學(xué)報;2013年01期

7 張書彬;韓冀中;劉志勇;王凱;;基于MapReduce實現(xiàn)空間查詢的研究[J];高技術(shù)通訊;2010年07期

8 劉凱;湯國安;江嶺;宋效東;陽建逸;張剛;;數(shù)字地形分析中鄰域統(tǒng)計型算法并行化方法及效率分析[J];地理與地理信息科學(xué);2013年04期

9 劉義;陳犖;景寧;熊偉;;基于R-樹索引的Map-Reduce空間連接聚集操作[J];國防科技大學(xué)學(xué)報;2013年01期

10 曾志;劉仁義;李先濤;張豐;包衛(wèi)正;;一種基于分塊的遙感影像并行處理機制[J];浙江大學(xué)學(xué)報(理學(xué)版);2012年02期

相關(guān)博士學(xué)位論文 前1條

1 康俊鋒;云計算環(huán)境下高分辨率遙感影像存儲與高效管理技術(shù)研究[D];浙江大學(xué);2011年

本文編號：1969373