本文選題：MapReduce + Hadoop��； 參考：《南京理工大學》2017年碩士論文

【摘要】：近年來大數(shù)據(jù)計算已成為研究熱點,Hadoop和Spark都是基于MapReduce框架的廣泛應用的大數(shù)據(jù)計算平臺,其性能主要取決于任務調(diào)度的優(yōu)劣。因此,基于MapReduce框架的Hadoop和Spark環(huán)境下任務調(diào)度算法的研究具有一定的理論價值和實際意義。本文重點研究:Hadoop環(huán)境下批處理作業(yè)調(diào)度算法和Spark環(huán)境下Web服務的資源分配方法。針對Hadoop環(huán)境下優(yōu)化最大完工時間的批處理作業(yè)調(diào)度問題,本文將該問題模型化為具有準備時間的兩階段混合流水作業(yè)調(diào)度問題,并基于DAG(Directed Acyclic Graph)模型提出啟發(fā)式算法 DAGEA(Directed Acyclic Graph Earliest Available)和DAGEF(Directed Acyclic Graph Earliest Finish)�，F(xiàn)有求解具有準備時間的兩階段混合流水作業(yè)調(diào)度的算法往往基于甘特圖構造,此方法無法有效考慮各作業(yè)的可調(diào)度范圍。不同于此,DAGEA、DAGEF基于DAG構造,通過DAG計算各作業(yè)的可調(diào)度范圍并合理調(diào)整作業(yè)的開始時間,從而有效提高算法的性能和效率。模擬實驗驗證了該結論。Spark計算基于內(nèi)存,而Hadoop計算基于磁盤。Spark目前資源分配考慮空余核數(shù)和內(nèi)存等大粒度資源,本文在Spark環(huán)境下Web服務資源調(diào)度增加考慮集群節(jié)點CPU利用率和處理能力等資源使用情況,重新評估每個節(jié)點資源利用率,再分配資源給任務。新的資源調(diào)度方法MEAN縮小資源粒度,從而提高集群資源利用率,增加Web請求處理數(shù),提高并發(fā)性。任務調(diào)度和資源分配是分布式大數(shù)據(jù)計算平臺的核心,其質量直接決定平臺的性能。本文研究基于MapReduce框架的任務調(diào)度算法,重點研究Hadoop環(huán)境下批處理調(diào)度算法和Spark環(huán)境下Web服務的資源分配方法,分別提出DAGEA、DAGEF和MEAN算法,實驗表明所提算法的有效性。
[Abstract]:In recent years, big data computing has become a hot research topic. Both Hadoop and Spark are widely used platforms based on MapReduce framework. The performance of big data computing platform mainly depends on the quality of task scheduling.Therefore, the research of task scheduling algorithm based on MapReduce framework in Hadoop and Spark environment has certain theoretical value and practical significance.This paper focuses on the task scheduling algorithm of batch processing under the environment of: Hadoop and the resource allocation method of Web service in Spark environment.Aiming at the batch scheduling problem which optimizes the maximum completion time in Hadoop environment, this paper models the problem as a two-stage mixed flow job scheduling problem with preparation time.A heuristic algorithm DAGEA(Directed Acyclic Graph Earliest available and DAGEF(Directed Acyclic Graph Earliest finish are proposed based on DAG(Directed Acyclic Graph model.The existing algorithms for solving two-stage mixed flow job scheduling with preparation time are often constructed based on Gantt graph. This method can not effectively consider the schedulable range of each job.Different from the DAG structure, the schedulable range of each job is calculated by DAG and the start time of the job is adjusted reasonably, so that the performance and efficiency of the algorithm can be improved effectively.The simulation results show that the Spark calculation is based on memory, while the Hadoop calculation is based on disk Spark's current resource allocation, which takes into account large granularity resources such as the number of spare cores and memory.In this paper, Web service resource scheduling in Spark environment takes into account the utilization of cluster nodes' CPU utilization and processing power, and reevaluates the utilization of each node's resources, and assigns the resources to the task.A new resource scheduling method, MEAN, reduces the granularity of resources, improves the utilization of cluster resources, increases the number of Web requests, and improves concurrency.Task scheduling and resource allocation are the core of the distributed big data computing platform, whose quality directly determines the performance of the platform.In this paper, the task scheduling algorithm based on MapReduce framework is studied, and the batch scheduling algorithm under Hadoop environment and the resource allocation method of Web service under Spark environment are studied. The DAGEAA DAGEF and MEAN algorithms are proposed, respectively. Experiments show that the proposed algorithm is effective.
【學位授予單位】：南京理工大學
【學位級別】：碩士
【學位授予年份】：2017
【分類號】：TP393.09;TP311.13

【相似文獻】

相關期刊論文 前10條

1 祿樂濱,劉明東;一種基于函數(shù)的多任務調(diào)度算法[J];空軍工程大學學報(自然科學版);2000年02期

2 阮幼林 ,劉干 ,朱光喜 ,盧小峰;一個基于復制的相關任務調(diào)度算法[J];小型微型計算機系統(tǒng);2005年03期

3 楊斌;張建軍;;一個新的基于通信競爭的任務調(diào)度算法[J];計算機工程與應用;2007年33期

4 胡同福;王文生;謝能付;;設備網(wǎng)格中的任務調(diào)度算法[J];計算機工程與設計;2008年12期

5 周艷慧;張凱;;新的分布式任務調(diào)度算法[J];計算機系統(tǒng)應用;2008年10期

6 薛繼偉;姜波;劉慶強;王征;;基于能力感知的人機任務調(diào)度算法[J];計算機工程;2009年19期

7 曹曉磊;程東年;黃萬偉;;基于離散時間距的在線可重構任務調(diào)度算法[J];小型微型計算機系統(tǒng);2010年10期

8 韓曉亞;汪斌強;黃萬偉;王保進;;采用配置完成優(yōu)先策略的可重構任務調(diào)度算法[J];小型微型計算機系統(tǒng);2012年03期

9 楊麗;武小年;商可e，

本文編號：1758417