本文選題：智能交通　切入點(diǎn)：大數(shù)據(jù)　出處：《杭州電子科技大學(xué)》2017年碩士論文　論文類型：學(xué)位論文

【摘要】：隨著中國城市化規(guī)模的逐步擴(kuò)大以及城市居民收入的不斷增加,私家車數(shù)量也在不斷的增加,伴隨著也帶來了一系列的交通問題。為了便于城市交通的管理,智能交通系統(tǒng)應(yīng)運(yùn)而生,其通過引入現(xiàn)代化技術(shù)并結(jié)合各城市的具體需求,實(shí)現(xiàn)對交通信息的實(shí)時收集和處理,了解當(dāng)前的交通環(huán)境并作出相應(yīng)的調(diào)控。這對于保障城市交通高效運(yùn)行和可持續(xù)發(fā)展具有重要的意義。數(shù)據(jù)存儲與檢索是智能交通系統(tǒng)的核心之一。在實(shí)際公安局點(diǎn)應(yīng)用中,道路監(jiān)控每天會產(chǎn)生海量的數(shù)據(jù),僅浙江省一天產(chǎn)生的過車數(shù)據(jù)便有幾億,并且數(shù)據(jù)到達(dá)具有隨機(jī)性。傳統(tǒng)的關(guān)系型數(shù)據(jù)庫由于其嚴(yán)格的表結(jié)構(gòu)約束,無法實(shí)現(xiàn)海量數(shù)據(jù)的存取操作。并且當(dāng)一張數(shù)據(jù)表的數(shù)據(jù)量達(dá)到一定級別時,索引本身就過于巨大。因此數(shù)據(jù)庫的檢索功能根本無法滿足數(shù)據(jù)檢索的需求,并且極易造成系統(tǒng)的癱瘓。本文對以上問題進(jìn)行深入研究,設(shè)計了千億級智能交通大數(shù)據(jù)存儲與檢索系統(tǒng)。系統(tǒng)采用分布式集群方案,以分布式框架Hadoop為基礎(chǔ)將系統(tǒng)集群設(shè)計為主從架構(gòu)。集群使用Zookeeper進(jìn)行一致性管理,使用Yarn進(jìn)行資源管理和分配。為保證系統(tǒng)集群的穩(wěn)定性,通過虛擬IP和Zookeeper實(shí)現(xiàn)了負(fù)載均衡和高可用性機(jī)制,用于處理高并發(fā)連接和單點(diǎn)故障問題,并保證對外地址的一致性。針對海量數(shù)據(jù)存儲和檢索這個難點(diǎn),引入搜索引擎Solr和非關(guān)系型數(shù)據(jù)庫HBase實(shí)現(xiàn)數(shù)據(jù)存儲和檢索方案。針對高并發(fā)數(shù)據(jù)容易引起Solr不穩(wěn)定,設(shè)計了Kafka和Spark Streaming高并發(fā)實(shí)時數(shù)據(jù)緩存和消費(fèi)策略。針對海量數(shù)據(jù)檢索延遲高,設(shè)計了自稱為Solr分Core算法和時間緊縮算法,實(shí)現(xiàn)了千億數(shù)據(jù)秒級檢索,并設(shè)計了翻頁緩存功能提升客戶端翻頁體驗(yàn)。最后,本文對系統(tǒng)進(jìn)行測試,測試結(jié)果表明系統(tǒng)工作穩(wěn)定,能高效存儲海量多種類型數(shù)據(jù)。當(dāng)數(shù)據(jù)庫中存儲一千億條過車記錄時,對此TB級別數(shù)據(jù)進(jìn)行各種條件的檢索均能在1s內(nèi)響應(yīng)。
[Abstract]:With the gradual expansion of the scale of urbanization in China and the increasing income of urban residents, the number of private cars is also increasing, accompanied by a series of traffic problems. In order to facilitate the management of urban traffic, Intelligent transportation system emerges as the times require. It can collect and process traffic information in real time by introducing modern technology and combining with the specific needs of each city. Understand the current traffic environment and make corresponding regulation and control. This is of great significance for ensuring the efficient operation and sustainable development of urban traffic. Data storage and retrieval is one of the core of intelligent transportation system. Road monitoring produces huge amounts of data every day. In Zhejiang Province alone, hundreds of millions of traffic data are generated in one day, and the arrival of the data is random. Traditional relational databases are constrained by their strict table structure. When the data amount of a data table reaches a certain level, the index itself is too large. Therefore, the retrieval function of the database can not meet the requirements of data retrieval at all. And it is easy to cause paralysis of the system. This paper makes a thorough study on the above problems, and designs a storage and retrieval system for big data, a 100bn level intelligent transportation system. The system adopts a distributed cluster scheme. The system cluster design is based on the distributed framework Hadoop. The cluster uses Zookeeper for consistency management and Yarn for resource management and allocation. A load balancing and high availability mechanism is implemented through virtual IP and Zookeeper, which is used to deal with the problems of high concurrent connection and single point failure, and to ensure the consistency of external address. This paper introduces search engine Solr and non-relational database HBase to realize the scheme of data storage and retrieval, aiming at the instability of Solr caused by high concurrency data. The high concurrent real-time data cache and consumption strategy of Kafka and Spark Streaming are designed. In view of the high latency of massive data retrieval, the self-called Solr sub-#en4# algorithm and the time compression algorithm are designed. Finally, the system is tested in this paper. The test results show that the system works stably and can store a large number of kinds of data efficiently. When 100 billion passing records are stored in the database, The terabyte level data can be retrieved in 1 s.
【學(xué)位授予單位】：杭州電子科技大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2017
【分類號】：U495;TP311.13

【參考文獻(xiàn)】

相關(guān)期刊論文 前10條

1 滕學(xué)蓓;李宏海;高劍;;中國智能交通系統(tǒng):要走自己的路[J];中國公路;2016年13期

2 王文峰;袁慶祝;陸佃龍;;基于Zookeeper綜合任務(wù)調(diào)度平臺的設(shè)計與應(yīng)用[J];信息技術(shù);2016年06期

3 黃力彬;;國內(nèi)外城市智能交通系統(tǒng)建設(shè)綜述[J];城市道橋與防洪;2016年05期

4 韓學(xué)仁;王青山;郭勇;;基于Solr的空間數(shù)據(jù)語義搜索研究[J];測繪與空間地理信息;2016年02期

5 葉駿宏;王勇;強(qiáng)保華;;基于Lucene的Web服務(wù)查詢系統(tǒng)設(shè)計與實(shí)現(xiàn)[J];桂林電子科技大學(xué)學(xué)報;2016年01期

6 王巖;王純;;一種基于Kafka的可靠的Consumer的設(shè)計方案[J];軟件;2016年01期

7 田翠珍;;基于Thrift框架RPC的研究與實(shí)現(xiàn)[J];信息與電腦(理論版);2016年01期

8 徐海榮;陳閔葉;張興媛;;基于Flume、Kafka、Storm、HDFS的航空維修大數(shù)據(jù)系統(tǒng)[J];上海工程技術(shù)大學(xué)學(xué)報;2015年04期

9 方金云;劉羽;姚曉;陳翠婷;張夢菲;肖茁建;張廣發(fā);;基于Spark的空間數(shù)據(jù)實(shí)時訪存技術(shù)的研究[J];地理信息世界;2015年06期

10 郝強(qiáng);高占春;;基于SolrCloud的網(wǎng)絡(luò)百科檢索服務(wù)的實(shí)現(xiàn)[J];軟件;2015年12期

相關(guān)碩士學(xué)位論文 前2條

1 王睿;基于主從服務(wù)器的分布式存儲系統(tǒng)的設(shè)計與實(shí)現(xiàn)[D];浙江大學(xué);2016年

2 莊共波;智能交通系統(tǒng)中的地圖路況服務(wù)研究與實(shí)現(xiàn)[D];北京郵電大學(xué);2013年

，

本文編號：1582429