本文關鍵詞： S4 檢查點技術 容錯計算 ZeroMQ 表決算法　出處：《北京郵電大學》2013年碩士論文　論文類型：學位論文

【摘要】：流計算是對實時數(shù)據(jù)流處理的研究,其理念是基于數(shù)據(jù)隱含價值隨著時間的流逝而降低,所以數(shù)據(jù)出現(xiàn)后需要盡快處理。而在各種解決方案中,分布式流計算框架因其具有很好的擴展性,易用性和靈活性而受到業(yè)界的普遍歡迎,本文即著手設計和實現(xiàn)分布式流計算框架中最為關鍵的容錯功能。 首先,本文立足于分布式流計算框架的研究背景和目前的研究走向,闡述容錯功能對分布式流計算框架的重要意義,并提出了設計和實現(xiàn)容錯功能的設想,介紹了實現(xiàn)分布式流計算框架容錯功能所需要的關鍵技術。 接下來,本文以開源的分布式流計算框架S4(Simple Scalable Stream System)為依托,詳細闡述分布式流計算框架的容錯功能需求分析,主要包含三大部分：處理節(jié)點容錯功能、處理節(jié)點間網(wǎng)絡通信容錯功能和處理節(jié)點計算容錯功能。處理節(jié)點容錯功能實現(xiàn)節(jié)點本身的可靠性,處理節(jié)點間網(wǎng)絡通常容錯功能實現(xiàn)網(wǎng)絡通信的可靠性,處理節(jié)點計算容錯功能實現(xiàn)應用程序處理結果的可靠性。 最后,本文對分布式流計算框架各容錯功能進行詳細設計和實現(xiàn)。處理節(jié)點容錯功能采用的是溫備份復制技術,定時的保存處理節(jié)點的狀態(tài)信息到第三方的數(shù)據(jù)庫MongoDB中,恢復處理節(jié)點時從MongoDB中獲取節(jié)點最近保存的狀態(tài)信息并初始化新創(chuàng)建的節(jié)點；處理節(jié)點間網(wǎng)絡通信容錯功能采用開源的網(wǎng)絡通信庫ZeroMQ實現(xiàn),ZeroMQ對消息差錯處理和網(wǎng)絡連接故障處理均具有較好的容錯能力；處理節(jié)點計算容錯功能采用多版本的軟件容錯技術屏蔽系統(tǒng)運行中可能發(fā)生的各種故障。論文末尾對主要的容錯功能模塊進行了相應測試和分析,指出容錯功能使分布式流計算系統(tǒng)具有較好的可靠性和健壯性,同時,簡要概括論文有待改進的方面以及下一步的工作。
[Abstract]:Stream computing is a research on real-time data stream processing. The idea is that the implicit value of data decreases with the passage of time, so the data needs to be processed as soon as possible after the emergence of data. Distributed flow computing framework has been widely welcomed by the industry because of its good scalability, ease of use and flexibility. This paper begins to design and implement the most critical fault-tolerant function in the distributed flow computing framework. First of all, based on the research background and current research trend of distributed flow computing framework, this paper expounds the importance of fault-tolerant function to distributed flow computing framework, and proposes the design and implementation of fault-tolerant function. This paper introduces the key technologies needed to implement the fault-tolerant function of distributed stream computing framework. Next, this paper relies on the open source distributed stream computing framework S4N simple Scalable Stream system. The requirement analysis of fault-tolerant function of distributed flow computing framework consists of three parts: processing node fault-tolerant function. The fault-tolerant function of the processing node realizes the reliability of the node itself, and the fault-tolerant function of the processing node usually realizes the reliability of the network communication. The processing node computing fault tolerant function realizes the reliability of the processing result of the application program. Finally, this paper designs and implements the fault-tolerant functions of distributed stream computing framework in detail. The processing node fault-tolerant function is based on warm backup replication technology. Periodically saves the status information of the processing node into the third-party database MongoDB. The newly created node is initialized by retrieving the state information recently saved by the node from the MongoDB while recovering the processing node. The fault-tolerant function of network communication between processing nodes adopts the open source network communication library ZeroMQ to implement ZeroMQ which has good fault-tolerant capability for both message error handling and network connection fault processing. Processing node computing fault-tolerant function adopts multi-version software fault-tolerant technology to shield all kinds of faults that may occur in the operation of the system. At the end of the paper the main fault-tolerant function modules are tested and analyzed accordingly. It is pointed out that the fault-tolerant function makes the distributed stream computing system more reliable and robust. At the same time, the paper briefly summarizes the aspects that need to be improved and the next work to be done.
【學位授予單位】：北京郵電大學
【學位級別】：碩士
【學位授予年份】：2013
【分類號】：TP302.8

【參考文獻】

相關期刊論文 前1條

1 高明;金澈清;王曉玲;田秀霞;周傲英;;數(shù)據(jù)世系管理技術研究綜述[J];計算機學報;2010年03期

相關博士學位論文 前1條

1 李磊;分布式系統(tǒng)中容錯機制性能優(yōu)化技術研究[D];國防科學技術大學;2007年

，

本文編號：1442515