【摘要】：隨著WEB 2.0,移動互聯(lián)網(wǎng)和物聯(lián)網(wǎng)的蓬勃發(fā)展等,使現(xiàn)在的數(shù)據(jù)呈現(xiàn)爆炸性的增長。面向海量數(shù)據(jù)的大數(shù)據(jù)平臺研究和開發(fā)正在成為業(yè)界的熱點。數(shù)據(jù)的采集、存儲和分析是大數(shù)據(jù)平臺的核心問題,其中數(shù)據(jù)采集主要用以解決多樣數(shù)據(jù)源的接入問題;數(shù)據(jù)存儲主要用于解決數(shù)據(jù)格式多樣性和數(shù)據(jù)海量存儲等問題;而數(shù)據(jù)分析以提供多樣性的算法為目標(biāo),為此必須提供易插拔的分析算法引擎。本文基于對主流大數(shù)據(jù)開源框架的調(diào)研、分析和應(yīng)用,提出了一種基于分布式存儲的數(shù)據(jù)采集和分析平臺。該解決方案以統(tǒng)一接口實現(xiàn)數(shù)據(jù)采集,整合多種存儲方式提供數(shù)據(jù)存儲,以插件化的算法引擎提供多樣化的數(shù)據(jù)分析。本文從調(diào)研主流開源框架和技術(shù)平臺入手,分析了分布式系統(tǒng)的技術(shù)特點,提出了面向大數(shù)據(jù)的數(shù)據(jù)采集和分析平臺的功能性以及非功能性需求;接著介紹了本平臺的設(shè)計迭代過程,對平臺架構(gòu)演進(jìn)的各個階段都進(jìn)行了細(xì)致的說明;而后對本平臺的實現(xiàn)和測試進(jìn)行了詳細(xì)的闡述,給出了功能性和非功能性兩方面的測試結(jié)果;最后通過介紹了兩個具體的應(yīng)用實例,進(jìn)一步驗證了本平臺的有效性。
[Abstract]:With the rapid development of WEB 2.0, mobile Internet and Internet of things, the current data show explosive growth. The research and development of big data platform for massive data is becoming a hot spot in the industry. Data collection, storage and analysis are the core problems of big data platform, in which data acquisition is mainly used to solve the access problem of various data sources; data storage is mainly used to solve the problems of data format diversity and data mass storage; and data analysis aims at providing diverse algorithms, so it is necessary to provide an easy-to-plug analysis algorithm engine. Based on the investigation, analysis and application of the mainstream big data open source framework, this paper proposes a data acquisition and analysis platform based on distributed storage. The solution realizes data acquisition with unified interface, integrates a variety of storage methods to provide data storage, and provides diversified data analysis by plug-in algorithm engine. Starting with the investigation of the mainstream open source framework and technology platform, this paper analyzes the technical characteristics of the distributed system, puts forward the functional and non-functional requirements of the data acquisition and analysis platform for big data, then introduces the design iteration process of the platform, and explains in detail the various stages of the evolution of the platform architecture. Then the implementation and test of the platform are described in detail, and the test results of functional and non-functional are given. Finally, two specific application examples are introduced to further verify the effectiveness of the platform.
【學(xué)位授予單位】：北京郵電大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2016
【分類號】：TP274.2;TP311.52

【參考文獻(xiàn)】

相關(guān)期刊論文 前2條

1 徐鵬;陳思;蘇森;;互聯(lián)網(wǎng)應(yīng)用PaaS平臺體系結(jié)構(gòu)[J];北京郵電大學(xué)學(xué)報;2012年01期

2 徐晶,許煒;消息中間件綜述[J];計算機(jī)工程;2005年16期

，

本文編號：2499844