面向并行數(shù)字地形分析的DEM數(shù)據(jù)云存儲(chǔ)方法研究
發(fā)布時(shí)間：2018-10-20 08:45
【摘要】：人類對(duì)地觀測(cè)技術(shù)的發(fā)展使得大范圍高精度地形數(shù)字高程模型(DEM)數(shù)據(jù)的數(shù)據(jù)量呈爆炸式增長�；趩我挥�(jì)算機(jī)的傳統(tǒng)數(shù)字地形分析方法由于存儲(chǔ)與計(jì)算性能有限,無法滿足地學(xué)研究和生產(chǎn)應(yīng)用的需求。面向新型硬件架構(gòu)的并行數(shù)字地形分析理論和方法的研究已經(jīng)成為地學(xué)界的研究熱點(diǎn)之一。然而,目前的研究著重于考慮地形分析算法的并行,很少涉及面向并行數(shù)字地形分析的海量DEM數(shù)據(jù)以及數(shù)字地形分析結(jié)果的管理問題,已有的DEM數(shù)據(jù)管理模式也無法滿足并行數(shù)字地形分析的需求。分布式計(jì)算平臺(tái)Hadoop被認(rèn)為是海量數(shù)據(jù)處理和分析的利器。如何利用Hadoop來管理急劇增加的DEM數(shù)據(jù),以滿足并行數(shù)字地形分析的需求,是值得研究的課題,本文即圍繞此課題展開研究,主要內(nèi)容包括以下三個(gè)方面： 1)面向并行數(shù)字地形分析的DEM數(shù)據(jù)組織方式 在深入分析并行數(shù)字地形分析的特點(diǎn)及其對(duì)DEM數(shù)據(jù)組織和管理需求的基礎(chǔ)上,詳細(xì)介紹了DEM金字塔結(jié)構(gòu)及其構(gòu)建方法和數(shù)學(xué)模型,并基于此結(jié)構(gòu)設(shè)計(jì)了DEM及其增量數(shù)據(jù)的分布式存儲(chǔ)模型,可實(shí)現(xiàn)海量DEM數(shù)據(jù)及其并行數(shù)字地形分析結(jié)果的有效存儲(chǔ)。 21面向并行數(shù)字地形分析的DEM數(shù)據(jù)管理方法 DEM管理方法包括空間索引機(jī)制、數(shù)據(jù)壓縮方法、系統(tǒng)容錯(cuò)機(jī)制和并發(fā)訪問策略等。本文結(jié)合HBase數(shù)據(jù)庫的存儲(chǔ)模式,構(gòu)建了三級(jí)空間索引,并提出了基于內(nèi)容的空間索引方法,使系統(tǒng)同時(shí)支持基于特征查詢和基于內(nèi)容查詢。在數(shù)據(jù)壓縮方面,設(shè)計(jì)了高程增量游程編碼壓縮算法,并對(duì)算法的適用性進(jìn)行了實(shí)驗(yàn)驗(yàn)證。此外,深入研究了Hadoop的容錯(cuò)方案和高并發(fā)訪問策略,并結(jié)合DEM數(shù)據(jù)的特點(diǎn),給出了配置參數(shù)的建議值以及部分實(shí)現(xiàn)代碼。 3)DEM云存儲(chǔ)原型系統(tǒng)的設(shè)計(jì)與實(shí)現(xiàn) 通過分析DEM云存儲(chǔ)系統(tǒng)的設(shè)計(jì)需求,給出了系統(tǒng)的詳細(xì)設(shè)計(jì)方案。并以三臺(tái)PC機(jī)搭建Hadoop集群模擬DEM云存儲(chǔ)環(huán)境,在該集群環(huán)境上開發(fā)了DEM云存儲(chǔ)原型系統(tǒng)。該系統(tǒng)支持四種查詢方式：①基于文件名查詢；②基于范圍和分辨率查詢；③基于范圍和格網(wǎng)數(shù)查詢；④基于內(nèi)容查詢。以全球SRTM3數(shù)據(jù)為實(shí)驗(yàn)數(shù)據(jù)對(duì)系統(tǒng)進(jìn)行驗(yàn)證。實(shí)驗(yàn)結(jié)果表明,數(shù)據(jù)檢索的結(jié)果是完全正確的,在現(xiàn)有的集群環(huán)境條件下,數(shù)據(jù)檢索的時(shí)間效率是令人滿意的。
[Abstract]:With the development of human earth observation technology, the data volume of (DEM) data of large range and high precision terrain digital elevation model increases explosively. The traditional digital terrain analysis method based on a single computer is unable to meet the requirements of geoscience research and production applications because of its limited storage and computing performance. The research of parallel digital terrain analysis theory and method for new hardware architecture has become one of the research hotspots in geoscience. However, the current research focuses on considering the parallelism of terrain analysis algorithms, and rarely involves the massive DEM data for parallel digital terrain analysis and the management of digital terrain analysis results. The existing DEM data management model can not meet the needs of parallel digital terrain analysis. Hadoop, a distributed computing platform, is considered to be a powerful tool for mass data processing and analysis. How to use Hadoop to manage the rapidly increasing DEM data to meet the needs of parallel digital terrain analysis is a topic worthy of study. The main contents include the following three aspects: 1) the DEM data organization for parallel digital terrain analysis is based on the in-depth analysis of the characteristics of parallel digital terrain analysis and the requirements for DEM data organization and management. This paper introduces the pyramid structure of DEM, its construction method and mathematical model in detail, and designs the distributed storage model of DEM and its incremental data based on this structure. The efficient storage of massive DEM data and its parallel digital terrain analysis results can be realized. 21 the DEM data management method for parallel digital terrain analysis includes spatial index mechanism, data compression method, DEM management method. System fault tolerant mechanism and concurrent access strategy. Combined with the storage mode of HBase database, this paper constructs a three-level spatial index, and proposes a content-based spatial index method, which enables the system to support both feature-based query and content-based query. In the aspect of data compression, the algorithm of height increment run-length coding compression is designed, and the applicability of the algorithm is verified by experiments. In addition, the fault-tolerant scheme and high concurrency access strategy of Hadoop are deeply studied, and the characteristics of DEM data are combined. The design and implementation of DEM cloud storage prototype system are given. By analyzing the design requirements of DEM cloud storage system, the detailed design scheme of the system is given. The Hadoop cluster simulation DEM cloud storage environment is built with three PC computers, and the DEM cloud storage prototype system is developed in the cluster environment. The system supports four query modes: (1) file name query; (2) range and resolution based query; (3) range and mesh query; (4) content based query. The global SRTM3 data are used as experimental data to verify the system. The experimental results show that the results of data retrieval are correct and the time efficiency of data retrieval is satisfactory under the existing cluster environment.
【學(xué)位授予單位】：南京師范大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2013
【分類號(hào)】：P208
本文編號(hào)：2282643
[Abstract]:With the development of human earth observation technology, the data volume of (DEM) data of large range and high precision terrain digital elevation model increases explosively. The traditional digital terrain analysis method based on a single computer is unable to meet the requirements of geoscience research and production applications because of its limited storage and computing performance. The research of parallel digital terrain analysis theory and method for new hardware architecture has become one of the research hotspots in geoscience. However, the current research focuses on considering the parallelism of terrain analysis algorithms, and rarely involves the massive DEM data for parallel digital terrain analysis and the management of digital terrain analysis results. The existing DEM data management model can not meet the needs of parallel digital terrain analysis. Hadoop, a distributed computing platform, is considered to be a powerful tool for mass data processing and analysis. How to use Hadoop to manage the rapidly increasing DEM data to meet the needs of parallel digital terrain analysis is a topic worthy of study. The main contents include the following three aspects: 1) the DEM data organization for parallel digital terrain analysis is based on the in-depth analysis of the characteristics of parallel digital terrain analysis and the requirements for DEM data organization and management. This paper introduces the pyramid structure of DEM, its construction method and mathematical model in detail, and designs the distributed storage model of DEM and its incremental data based on this structure. The efficient storage of massive DEM data and its parallel digital terrain analysis results can be realized. 21 the DEM data management method for parallel digital terrain analysis includes spatial index mechanism, data compression method, DEM management method. System fault tolerant mechanism and concurrent access strategy. Combined with the storage mode of HBase database, this paper constructs a three-level spatial index, and proposes a content-based spatial index method, which enables the system to support both feature-based query and content-based query. In the aspect of data compression, the algorithm of height increment run-length coding compression is designed, and the applicability of the algorithm is verified by experiments. In addition, the fault-tolerant scheme and high concurrency access strategy of Hadoop are deeply studied, and the characteristics of DEM data are combined. The design and implementation of DEM cloud storage prototype system are given. By analyzing the design requirements of DEM cloud storage system, the detailed design scheme of the system is given. The Hadoop cluster simulation DEM cloud storage environment is built with three PC computers, and the DEM cloud storage prototype system is developed in the cluster environment. The system supports four query modes: (1) file name query; (2) range and resolution based query; (3) range and mesh query; (4) content based query. The global SRTM3 data are used as experimental data to verify the system. The experimental results show that the results of data retrieval are correct and the time efficiency of data retrieval is satisfactory under the existing cluster environment.
【學(xué)位授予單位】：南京師范大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2013
【分類號(hào)】：P208
【參考文獻(xiàn)】
相關(guān)期刊論文 前6條
1 覃雄派;王會(huì)舉;杜小勇;王珊;;大數(shù)據(jù)分析——RDBMS與MapReduce的競(jìng)爭(zhēng)與共生[J];軟件學(xué)報(bào);2012年01期
2 方濤,李德仁,龔健雅,皮明紅;GeoImageDB多分辨率無縫影像數(shù)據(jù)庫系統(tǒng)的開發(fā)與實(shí)現(xiàn)[J];武漢測(cè)繪科技大學(xué)學(xué)報(bào);1999年03期
3 陳仁喜;趙忠明;王殷行;;基于整型小波變換的DEM數(shù)據(jù)壓縮[J];武漢大學(xué)學(xué)報(bào)(信息科學(xué)版);2006年04期
4 多雪松;張晶;高強(qiáng);;基于Hadoop的海量數(shù)據(jù)管理系統(tǒng)[J];微計(jì)算機(jī)信息;2010年13期
5 鄭晶晶;方金云;韓承德;;一種快速的DEM數(shù)據(jù)無損壓縮算法[J];系統(tǒng)仿真學(xué)報(bào);2010年10期
6 趙立成,王素娟,施進(jìn)明;國家衛(wèi)星氣象中心信息共享體制研究與技術(shù)實(shí)現(xiàn)[J];應(yīng)用氣象學(xué)報(bào);2002年05期
，本文編號(hào)：2282643
本文鏈接：http://www.lk138.cn/kejilunwen/dizhicehuilunwen/2282643.html
最近更新
教材專著
