本文選題：大數(shù)據(jù)　切入點(diǎn)：分布式計(jì)算　出處：《華中科技大學(xué)》2012年碩士論文

【摘要】：隨著計(jì)算機(jī)和互聯(lián)網(wǎng)的發(fā)展，信息量的增長(zhǎng)已經(jīng)達(dá)到了一個(gè)前所未有的速度，海量數(shù)據(jù)的日益累積，使人們迎來(lái)了 大數(shù)據(jù)‖的時(shí)代。如何正確且高效地從這些數(shù)據(jù)中分析出有用信息并做出處理成為了一項(xiàng)極具挑戰(zhàn)性的工作。面對(duì)這一問(wèn)題，人們往往考慮采用分布式計(jì)算進(jìn)行處理。然而對(duì)于大多數(shù)信息企業(yè)來(lái)說(shuō)，往往存在著許多種不同需求的業(yè)務(wù)，這些業(yè)務(wù)所適用的分布式計(jì)算平臺(tái)不盡相同，這就導(dǎo)致了企業(yè)分布式計(jì)算平臺(tái)種類(lèi)的繁多，學(xué)習(xí)和維護(hù)這些平臺(tái)往往需要很高的成本，這給新型互聯(lián)網(wǎng)企業(yè)帶來(lái)了巨大的經(jīng)濟(jì)壓力。因此，設(shè)計(jì)一個(gè)支持多計(jì)算模型的分布式計(jì)算系統(tǒng)成為了各企業(yè)的迫切需求。 Antnest將企業(yè)的各種基本業(yè)務(wù)需求抽象成三種不同的計(jì)算模型，，通過(guò)對(duì)比這些計(jì)算模型的特點(diǎn)，對(duì)它們的可等價(jià)性進(jìn)行了分析，并提出了一種統(tǒng)一計(jì)算模型的方案，主要包括數(shù)據(jù)源的統(tǒng)一以及計(jì)算處理的統(tǒng)一。Antnest綜合了批處理分布式計(jì)算、流式分布式計(jì)算、任務(wù)自動(dòng)調(diào)度框架等三種平臺(tái)的特點(diǎn)，對(duì)企業(yè)中最基本的三種業(yè)務(wù)——大規(guī)模靜態(tài)數(shù)據(jù)處理業(yè)務(wù)、線(xiàn)上實(shí)時(shí)業(yè)務(wù)、后臺(tái)定時(shí)業(yè)務(wù)提供了很好的支持，從而減少了企業(yè)因使用和維護(hù)多種平臺(tái)所造成的開(kāi)銷(xiāo)。Antnest為用戶(hù)提供了一個(gè)統(tǒng)一的任務(wù)創(chuàng)建模式，用戶(hù)根據(jù)自己的實(shí)際業(yè)務(wù)需求采用相同的規(guī)則定義數(shù)據(jù)源及計(jì)算單元配置文件，并在對(duì)應(yīng)的任務(wù)接口中實(shí)現(xiàn)計(jì)算單元的處理邏輯，接著便能將任務(wù)上傳到該平臺(tái)中執(zhí)行。Antnest屏蔽了不同任務(wù)計(jì)算模型之間的差異。另外，底層的一些實(shí)現(xiàn)機(jī)制，如消息通信、心跳檢測(cè)、容錯(cuò)、任務(wù)的劃分及調(diào)度等，對(duì)于用戶(hù)而言是完全透明的。 經(jīng)測(cè)試表明，Antnest能很好的支持不同計(jì)算模型任務(wù)，正確執(zhí)行任務(wù)的各種基本操作，同時(shí)還對(duì)不同的數(shù)據(jù)庫(kù)提供了支持。
[Abstract]:With the development of computer and Internet, the increase of information has reached an unprecedented speed, and the accumulation of massive data makes people usher in the era of big data.How to correctly and efficiently analyze and process useful information from these data has become a challenging task.Faced with this problem, people often consider using distributed computing for processing.However, for most information enterprises, there are many kinds of business with different requirements, and the distributed computing platforms for these services are not the same, which leads to a great variety of distributed computing platforms.Learning and maintaining these platforms often require high cost, which brings great economic pressure to new Internet enterprises.Therefore, the design of a distributed computing system supporting multiple computing models has become an urgent need of enterprises.Antnest abstracts various basic business requirements into three different computing models. By comparing the characteristics of these models, the equivalence of these models is analyzed, and a unified computing model is proposed.It mainly includes the unification of data sources and the unification of computing processing. Antnest synthesizes the characteristics of three kinds of platforms, such as batch distributed computing, flow distributed computing, task automatic scheduling framework, etc.It provides very good support for the most basic three kinds of business in the enterprise-large-scale static data processing service, on-line real-time service, backstage timing service, etc.This reduces the cost of using and maintaining multiple platforms in enterprises. Antnest provides a unified task creation model for users. Users define data sources and cell configuration files according to their actual business requirements with the same rules.The processing logic of the computing unit is implemented in the corresponding task interface, and then the task can be uploaded to the platform to execute .Antnest to shield the differences between different task computing models.In addition, the underlying implementation mechanisms, such as message communication, heartbeat detection, fault tolerance, task partitioning and scheduling, are completely transparent to users.The test results show that Antnest can support different computing model tasks, perform all kinds of basic operations correctly, and provide support for different databases.
【學(xué)位授予單位】：華中科技大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2012
【分類(lèi)號(hào)】：TP338.8

【參考文獻(xiàn)】

相關(guān)期刊論文 前10條

1 王];董志勇;;基于Quartz的網(wǎng)管系統(tǒng)任務(wù)調(diào)度的實(shí)現(xiàn)[J];電腦開(kāi)發(fā)與應(yīng)用;2011年07期

2 王峰;雷葆華;;Hadoop分布式文件系統(tǒng)的模型分析[J];電信科學(xué);2010年12期

3 周園春,李淼,張建,李曉歐,張飛;中間件技術(shù)綜述[J];計(jì)算機(jī)工程與應(yīng)用;2002年15期

4 陳誠(chéng);陳海濤;;一種自適應(yīng)的容災(zāi)系統(tǒng)心跳檢測(cè)算法[J];計(jì)算機(jī)工程與科學(xué);2008年05期

5 蔣雄偉,馬范援;中間件與分布式計(jì)算[J];計(jì)算機(jī)應(yīng)用;2002年04期

6 白云川;;迎接大數(shù)據(jù)時(shí)代[J];中國(guó)制造業(yè)信息化;2011年12期

7 張敦華,劉建;Java動(dòng)態(tài)類(lèi)加載機(jī)制及其應(yīng)用[J];計(jì)算機(jī)工程與設(shè)計(jì);2004年03期

8 趙春雷;喬治·納漢;;“大數(shù)據(jù)”時(shí)代的計(jì)算機(jī)信息處理技術(shù)[J];世界科學(xué);2012年02期

9 封瑋,周世平;基于Java NIO的非阻塞通信的研究與實(shí)現(xiàn)[J];計(jì)算機(jī)系統(tǒng)應(yīng)用;2004年09期

10 金玉軍;姜曉彤;況迎輝;;基于高性能NIO的MINA框架的應(yīng)用[J];中國(guó)科技信息;2009年16期

相關(guān)碩士學(xué)位論文 前2條

1 張文峰;基于MapReduce模型的分布式計(jì)算平臺(tái)的原理與設(shè)計(jì)[D];華中科技大學(xué);2010年

2 張密密;MapReduce模型在Hadoop實(shí)現(xiàn)中的性能分析及改進(jìn)優(yōu)化[D];電子科技大學(xué);2010年

本文編號(hào)：1718852