發(fā)布時間：2019-01-01 17:38

【摘要】：隨著物聯(lián)網(wǎng)技術(shù)的發(fā)展,智能化傳感器的增多,交通行業(yè)收集到的數(shù)據(jù)急速增長。特別是在高速公路收費系統(tǒng)中,每天都會產(chǎn)生海量的高速公路收費站數(shù)據(jù)。通過分析這些結(jié)構(gòu)化的數(shù)據(jù),可以得到高速公路車流量、載運量時空分布、高速公路運輸景氣指數(shù)、收費報表同比環(huán)比等非常有價值的信息,為高速公路管理人員的正確決策提供數(shù)據(jù)支持。當前,大多數(shù)交通部門所使用的管理系統(tǒng)都是使用Oracle驅(qū)動的數(shù)據(jù)庫。面對數(shù)據(jù)體量愈發(fā)龐大的高速公路收費站數(shù)據(jù),這些管理系統(tǒng)已經(jīng)出現(xiàn)數(shù)據(jù)整合過程復雜、時間久、依賴專業(yè)人員、數(shù)據(jù)查詢速度慢等問題。因此,本文研究基于云平臺的高速公路交通數(shù)據(jù)倉庫設(shè)計與查詢優(yōu)化技術(shù)。首先,本文針對高速公路收費站數(shù)據(jù)特點,設(shè)計一種面向海量高速公路收費站數(shù)據(jù)的數(shù)據(jù)倉庫,其構(gòu)建過程包括數(shù)據(jù)抽取、數(shù)據(jù)預(yù)處理和數(shù)據(jù)加工等三個核心操作階段。其次,本文通過比較Hive和Impala的查詢特點,分析數(shù)據(jù)倉庫的分區(qū)粒度和高速公路管理的業(yè)務(wù)特點,提出了三種數(shù)據(jù)倉庫查詢優(yōu)化方法。然后,本文基于分布式文件存儲系統(tǒng)HDFS、數(shù)據(jù)倉庫工具Hive和數(shù)據(jù)查詢引擎Impala實現(xiàn)數(shù)據(jù)倉庫構(gòu)建,設(shè)計并實現(xiàn)了面向高速公路管理的數(shù)據(jù)可視化平臺,提供數(shù)據(jù)查詢及專題分析等功能。最后,本文使用實際的高速公路收費站數(shù)據(jù)驗證數(shù)據(jù)倉庫的功能和性能,結(jié)果表明本文提出的數(shù)據(jù)查詢優(yōu)化方法能夠有效提高數(shù)據(jù)查詢效率,縮短查詢時間。
[Abstract]:With the development of Internet of things technology and the increase of intelligent sensors, the data collected by transportation industry is increasing rapidly. Especially in the freeway toll collection system, a large amount of highway toll collection station data are generated every day. By analyzing these structured data, we can get very valuable information such as freeway traffic flow, space-time distribution of carrying capacity, expressway transportation boom index, toll report forms, and so on. Provide data support for highway managers to make correct decisions. Currently, most management systems used by transportation departments are Oracle-driven databases. Faced with the increasingly large data volume of highway toll station data, these management systems have problems such as complex data integration process, long time, dependence on professionals, slow data query speed and so on. Therefore, this paper studies the highway traffic data warehouse design and query optimization technology based on cloud platform. Firstly, according to the characteristics of highway toll station data, this paper designs a data warehouse for mass highway toll station data. The construction process includes three core operation stages: data extraction, data preprocessing and data processing. Secondly, by comparing the query characteristics of Hive and Impala, this paper analyzes the partition granularity of data warehouse and the business characteristics of highway management, and puts forward three query optimization methods of data warehouse. Then, based on the distributed file storage system HDFS, data warehouse tool Hive and the data query engine Impala, this paper designs and implements the data visualization platform for highway management. Provides data query and project analysis functions. Finally, the function and performance of the data warehouse are verified by the actual toll station data in this paper. The results show that the data query optimization method proposed in this paper can effectively improve the efficiency of data query and shorten the query time.
【學位授予單位】：北京郵電大學
【學位級別】：碩士
【學位授予年份】：2017
【分類號】：TP311.13;TP393.09

【參考文獻】

相關(guān)期刊論文 前7條

1 吳黎兵;邱鑫;葉璐瑤;王曉棟;聶雷;;基于Hadoop的SQL查詢引擎性能研究[J];華中師范大學學報(自然科學版);2016年02期

2 趙文英;;當前大數(shù)據(jù)管理技術(shù)探究[J];信息與電腦(理論版);2015年22期

3 曾萍;韋杰;;數(shù)據(jù)倉庫技術(shù)在高校信息化建設(shè)中的應(yīng)用研究[J];軟件;2014年05期

4 李小強;何珊;何金明;;通過對比數(shù)據(jù)庫來理解數(shù)據(jù)倉庫[J];考試周刊;2013年91期

5 邱衛(wèi)云;;智能交通大數(shù)據(jù)分析云平臺技術(shù)[J];中國交通信息化;2013年10期

6 黃文依;王勁松;林勝;;HDFS可視化操作研究與實現(xiàn)[J];天津理工大學學報;2012年01期

7 許春玲;張廣泉;;分布式文件系統(tǒng)Hadoop HDFS與傳統(tǒng)文件系統(tǒng)Linux FS的比較與分析[J];蘇州大學學報(工科版);2010年04期

相關(guān)碩士學位論文 前5條

1 張鵬;多數(shù)據(jù)庫環(huán)境數(shù)據(jù)集成與轉(zhuǎn)換技術(shù)研究[D];北方工業(yè)大學;2016年

2 費仕憶;Hadoop大數(shù)據(jù)平臺與傳統(tǒng)數(shù)據(jù)倉庫的協(xié)作研究[D];東華大學;2014年

3 王遠志;基于Hadoop的全網(wǎng)絡(luò)流量異常監(jiān)測算法研究[D];鄭州大學;2014年

4 韓歡;基于大數(shù)據(jù)的智能交通運輸平臺的研究[D];成都理工大學;2014年

5 常濤;改進型MapReduce框架的研究與設(shè)計[D];北京郵電大學;2011年

，

本文編號：2397894