本文關(guān)鍵詞：基于分布式框架的網(wǎng)絡(luò)事件實(shí)時感知系統(tǒng)　出處：《浙江大學(xué)》2017年碩士論文　論文類型：學(xué)位論文

  更多相關(guān)文章： 事件感知 分布式 實(shí)時 流式計算

【摘要】：隨著互聯(lián)網(wǎng)的發(fā)展,面對海量數(shù)據(jù)時,個人的精力無法滿足完成提取、獲得全面而精確的信息的任務(wù)的要求,從而掌握一個特定領(lǐng)域下的趨勢發(fā)展。基于此便提出了以事件形式作為載體,通過從不斷處理的新的文檔中提取事件信息,之后合并到舊有信息中,呈現(xiàn)給用戶宏觀上的統(tǒng)計數(shù)據(jù)和具體分析內(nèi)容,并輔助人們進(jìn)行各類決策�，F(xiàn)階段較為成熟的事件感知系統(tǒng)依賴于大規(guī)模計算集群,以流式與批量式集合的方式,完成了大數(shù)據(jù)規(guī)模下應(yīng)用的實(shí)現(xiàn)。本文聚焦于在小規(guī)模集群下能夠?qū)崟r獲取事件結(jié)果,進(jìn)行查詢的總體要求,以流式處理的形式,在增加系統(tǒng)整體處理效率與減少對算法影響的目標(biāo)下,完成事件感知各項應(yīng)用功能。本文基于上述目標(biāo),設(shè)計并開發(fā)了一套分布式處理平臺,滿足應(yīng)用在各個環(huán)節(jié)下的應(yīng)用要求。主要的工作包括:1)針對事件感知應(yīng)用的輸入、輸出,用戶對象進(jìn)行分析,將系統(tǒng)劃分為三個模塊,完成系統(tǒng)總體架構(gòu)設(shè)計。2)在存儲模塊下設(shè)計了存儲形式,包括MongoDB內(nèi)數(shù)據(jù)的表達(dá)與NAF標(biāo)引格式。3)在處理模塊下,對事件感知傳統(tǒng)的兩種類型任務(wù)在流式數(shù)據(jù)環(huán)境下進(jìn)行了分布式擴(kuò)展,提出了各自的拓?fù)湓O(shè)計。同時針對系統(tǒng)運(yùn)行的Storm計算框架,優(yōu)化了拓?fù)湔{(diào)度器,并針對內(nèi)存計算設(shè)計了符合事件感知容錯性要求的內(nèi)存數(shù)據(jù)的持久化策略。4)分析與服務(wù)模塊設(shè)計了針對不同查詢類型的響應(yīng)策略,并在查詢后臺設(shè)計了在分布式內(nèi)存環(huán)境下基于封閉立方體的維度統(tǒng)計方法最后以實(shí)際檢驗檢疫應(yīng)用出發(fā)為導(dǎo)向,驗證了系統(tǒng)的可用性與性能。
[Abstract]:With the development of the Internet, in the face of massive data, the individual energy can not meet the task of extracting, obtaining comprehensive and accurate information. In order to grasp the trend of development in a specific field. Based on this, it is proposed to take the form of events as the carrier, through the continuous processing of new documents from the extraction of event information, and then merged into the old information. It presents users with macroscopic statistical data and concrete analysis content, and assists people to make all kinds of decisions. At this stage, the more mature event perception system relies on large-scale computing clusters. The implementation of big data application under the scale of big data is completed by the way of flow and batch collection. This paper focuses on the overall requirements of real-time event results and query in small scale cluster, in the form of flow processing. Under the goal of increasing the overall processing efficiency of the system and reducing the impact on the algorithm, this paper designs and develops a distributed processing platform based on the above objectives. The main work includes: 1) analyzing the input, output and user object of the event-aware application, and dividing the system into three modules. Complete the system architecture design. 2) Design the storage form under the storage module, including the data expression in MongoDB and the NAF indexing format. 3) under the processing module. Two kinds of traditional event-aware tasks are extended in the streaming data environment, and their topology design is proposed. At the same time, the topology scheduler is optimized for the Storm computing framework. The persistence strategy of memory data, which meets the requirements of event-aware fault-tolerance, is designed for memory computing. 4) Analysis and service modules are designed to respond to different query types. The dimension statistics method based on closed cube in distributed memory environment is designed in the query background. Finally, the application of practical inspection and quarantine is taken as the guide to verify the availability and performance of the system.
【學(xué)位授予單位】：浙江大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2017
【分類號】：TP311.13

【參考文獻(xiàn)】

相關(guān)期刊論文 前4條

1 張亮;白振興;周軍;白云;;一種生成封閉數(shù)據(jù)立方體的新算法[J];彈箭與制導(dǎo)學(xué)報;2010年03期

2 吳飛;莊越挺;;互聯(lián)網(wǎng)跨媒體分析與檢索:理論與算法[J];計算機(jī)輔助設(shè)計與圖形學(xué)學(xué)報;2010年01期

3 游進(jìn)國;奚建清;張平健;劉艷霞;;在PC集群上的封閉立方體計算[J];計算機(jī)科學(xué);2009年06期

4 李盛恩,王珊;封閉數(shù)據(jù)立方體技術(shù)研究[J];軟件學(xué)報;2004年08期

，

本文編號：1407872