【摘要】：大數(shù)據(jù)時(shí)代的來臨,為數(shù)據(jù)的實(shí)時(shí)處理技術(shù)帶來了巨大的變革和挑戰(zhàn),在這個(gè)背景下,D-Stream作為D-Ocean非結(jié)構(gòu)化數(shù)據(jù)管理系統(tǒng)的流式處理子系統(tǒng),為基于海量數(shù)據(jù)實(shí)時(shí)處理的應(yīng)用提供給了一套通用的、可靠的、可擴(kuò)展的分布式計(jì)算框架。 D-Stream系統(tǒng)的實(shí)現(xiàn)基于一套流式處理通用框架設(shè)計(jì),借鑒了S4, Storm等眾多開源流式處理平臺(tái)的先進(jìn)思想。它的功能結(jié)構(gòu)主要包含三大部分：首先是一套簡(jiǎn)潔開放的任務(wù)模型,通過D-Stream任務(wù)模型,應(yīng)用能夠根據(jù)需求動(dòng)態(tài)地定制任務(wù)拓?fù)洹Ｆ浯问且惶卓煽糠�(wěn)定的流式處理引擎,保障數(shù)據(jù)在計(jì)算任務(wù)間快速透明地傳輸,使不同的任務(wù)可以高效地協(xié)調(diào)工作。最后是一個(gè)高可用的調(diào)度框架,通過有效調(diào)度計(jì)算資源,充分發(fā)揮集群的計(jì)算能力,并在消息積壓時(shí)提供高效的負(fù)載分流機(jī)制。 圍繞這三大問題,本文描述了用D-Stream任務(wù)模型為現(xiàn)實(shí)應(yīng)用建模的方法。就D-Stream實(shí)現(xiàn)部分,介紹了D-Stream組件架構(gòu)和對(duì)稱式調(diào)度框架,重點(diǎn)描述了流式處理引擎實(shí)現(xiàn)中用到的相關(guān)算法和設(shè)計(jì)模式,并針對(duì)D-Stream系統(tǒng)的各方面特性給予了全面評(píng)估。最后本文通過D-Ocean CBIR勺應(yīng)用案例,驗(yàn)證了在海量數(shù)據(jù)的實(shí)時(shí)應(yīng)用中D-Stream系統(tǒng)的優(yōu)越性。
[Abstract]:The advent of big data era has brought great changes and challenges to the real-time data processing technology. Under this background, D-Stream is the flow processing subsystem of D-Ocean unstructured data management system. This paper presents a general, reliable and extensible distributed computing framework for applications based on real-time processing of mass data. The implementation of D-Stream system is based on a set of general frame design of flow processing, and the advanced ideas of many open source streaming processing platforms such as S4, Storm and so on are used for reference. Its functional structure mainly includes three parts: first, a set of simple and open task model. Through the D-Stream task model, the application can dynamically customize the task topology according to the requirements. The second is a reliable and stable flow engine, which ensures the fast and transparent transmission of data between computing tasks, so that different tasks can be coordinated efficiently. Finally, it is a highly available scheduling framework, which can effectively schedule computing resources, give full play to the computing power of the cluster, and provide an efficient load shunt mechanism in the case of message backlog. Around these three problems, this paper describes the method of using D-Stream task model to model practical applications. In the part of D-Stream implementation, the D-Stream component architecture and the symmetric scheduling framework are introduced, and the relevant algorithms and design patterns used in the implementation of the flow processing engine are described, and the overall evaluation of the characteristics of the D-Stream system is given. Finally, this paper verifies the superiority of D-Stream system in the real time application of massive data through the application case of D-Ocean CBIR spoon.
【學(xué)位授予單位】：浙江大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2013
【分類號(hào)】：TP338.8

【參考文獻(xiàn)】

相關(guān)期刊論文 前1條

1 周琪;;大數(shù)據(jù)時(shí)代來臨,你準(zhǔn)備好沒有?[J];電腦與電信;2012年08期

，

本文編號(hào)：2376847