本文關(guān)鍵詞： S4 檢查點(diǎn)技術(shù) 容錯(cuò)計(jì)算 ZeroMQ 表決算法　出處：《北京郵電大學(xué)》2013年碩士論文　論文類(lèi)型：學(xué)位論文

【摘要】：流計(jì)算是對(duì)實(shí)時(shí)數(shù)據(jù)流處理的研究,其理念是基于數(shù)據(jù)隱含價(jià)值隨著時(shí)間的流逝而降低,所以數(shù)據(jù)出現(xiàn)后需要盡快處理。而在各種解決方案中,分布式流計(jì)算框架因其具有很好的擴(kuò)展性,易用性和靈活性而受到業(yè)界的普遍歡迎,本文即著手設(shè)計(jì)和實(shí)現(xiàn)分布式流計(jì)算框架中最為關(guān)鍵的容錯(cuò)功能。 首先,本文立足于分布式流計(jì)算框架的研究背景和目前的研究走向,闡述容錯(cuò)功能對(duì)分布式流計(jì)算框架的重要意義,并提出了設(shè)計(jì)和實(shí)現(xiàn)容錯(cuò)功能的設(shè)想,介紹了實(shí)現(xiàn)分布式流計(jì)算框架容錯(cuò)功能所需要的關(guān)鍵技術(shù)。 接下來(lái),本文以開(kāi)源的分布式流計(jì)算框架S4(Simple Scalable Stream System)為依托,詳細(xì)闡述分布式流計(jì)算框架的容錯(cuò)功能需求分析,主要包含三大部分：處理節(jié)點(diǎn)容錯(cuò)功能、處理節(jié)點(diǎn)間網(wǎng)絡(luò)通信容錯(cuò)功能和處理節(jié)點(diǎn)計(jì)算容錯(cuò)功能。處理節(jié)點(diǎn)容錯(cuò)功能實(shí)現(xiàn)節(jié)點(diǎn)本身的可靠性,處理節(jié)點(diǎn)間網(wǎng)絡(luò)通常容錯(cuò)功能實(shí)現(xiàn)網(wǎng)絡(luò)通信的可靠性,處理節(jié)點(diǎn)計(jì)算容錯(cuò)功能實(shí)現(xiàn)應(yīng)用程序處理結(jié)果的可靠性。 最后,本文對(duì)分布式流計(jì)算框架各容錯(cuò)功能進(jìn)行詳細(xì)設(shè)計(jì)和實(shí)現(xiàn)。處理節(jié)點(diǎn)容錯(cuò)功能采用的是溫備份復(fù)制技術(shù),定時(shí)的保存處理節(jié)點(diǎn)的狀態(tài)信息到第三方的數(shù)據(jù)庫(kù)MongoDB中,恢復(fù)處理節(jié)點(diǎn)時(shí)從MongoDB中獲取節(jié)點(diǎn)最近保存的狀態(tài)信息并初始化新創(chuàng)建的節(jié)點(diǎn)；處理節(jié)點(diǎn)間網(wǎng)絡(luò)通信容錯(cuò)功能采用開(kāi)源的網(wǎng)絡(luò)通信庫(kù)ZeroMQ實(shí)現(xiàn),ZeroMQ對(duì)消息差錯(cuò)處理和網(wǎng)絡(luò)連接故障處理均具有較好的容錯(cuò)能力；處理節(jié)點(diǎn)計(jì)算容錯(cuò)功能采用多版本的軟件容錯(cuò)技術(shù)屏蔽系統(tǒng)運(yùn)行中可能發(fā)生的各種故障。論文末尾對(duì)主要的容錯(cuò)功能模塊進(jìn)行了相應(yīng)測(cè)試和分析,指出容錯(cuò)功能使分布式流計(jì)算系統(tǒng)具有較好的可靠性和健壯性,同時(shí),簡(jiǎn)要概括論文有待改進(jìn)的方面以及下一步的工作。
[Abstract]:Stream computing is a research on real-time data stream processing. The idea is that the implicit value of data decreases with the passage of time, so the data needs to be processed as soon as possible after the emergence of data. Distributed flow computing framework has been widely welcomed by the industry because of its good scalability, ease of use and flexibility. This paper begins to design and implement the most critical fault-tolerant function in the distributed flow computing framework. First of all, based on the research background and current research trend of distributed flow computing framework, this paper expounds the importance of fault-tolerant function to distributed flow computing framework, and proposes the design and implementation of fault-tolerant function. This paper introduces the key technologies needed to implement the fault-tolerant function of distributed stream computing framework. Next, this paper relies on the open source distributed stream computing framework S4N simple Scalable Stream system. The requirement analysis of fault-tolerant function of distributed flow computing framework consists of three parts: processing node fault-tolerant function. The fault-tolerant function of the processing node realizes the reliability of the node itself, and the fault-tolerant function of the processing node usually realizes the reliability of the network communication. The processing node computing fault tolerant function realizes the reliability of the processing result of the application program. Finally, this paper designs and implements the fault-tolerant functions of distributed stream computing framework in detail. The processing node fault-tolerant function is based on warm backup replication technology. Periodically saves the status information of the processing node into the third-party database MongoDB. The newly created node is initialized by retrieving the state information recently saved by the node from the MongoDB while recovering the processing node. The fault-tolerant function of network communication between processing nodes adopts the open source network communication library ZeroMQ to implement ZeroMQ which has good fault-tolerant capability for both message error handling and network connection fault processing. Processing node computing fault-tolerant function adopts multi-version software fault-tolerant technology to shield all kinds of faults that may occur in the operation of the system. At the end of the paper the main fault-tolerant function modules are tested and analyzed accordingly. It is pointed out that the fault-tolerant function makes the distributed stream computing system more reliable and robust. At the same time, the paper briefly summarizes the aspects that need to be improved and the next work to be done.
【學(xué)位授予單位】：北京郵電大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2013
【分類(lèi)號(hào)】：TP302.8

【參考文獻(xiàn)】

相關(guān)期刊論文 前1條

1 高明;金澈清;王曉玲;田秀霞;周傲英;;數(shù)據(jù)世系管理技術(shù)研究綜述[J];計(jì)算機(jī)學(xué)報(bào);2010年03期

相關(guān)博士學(xué)位論文 前1條

1 李磊;分布式系統(tǒng)中容錯(cuò)機(jī)制性能優(yōu)化技術(shù)研究[D];國(guó)防科學(xué)技術(shù)大學(xué);2007年

，

本文編號(hào)：1442515