本文選題：可靠性評估 + 任務(wù)調(diào)度　； 參考：《北京郵電大學(xué)》2014年碩士論文

【摘要】：隨著人們對計(jì)算能力需求的逐漸增大,高性能計(jì)算技術(shù)的發(fā)展催生出云計(jì)算概念。作為新型的服務(wù)與計(jì)算模型,云計(jì)算的出現(xiàn)使得計(jì)算能力和存儲資源可以按需獲取,用戶服務(wù)與應(yīng)用通過網(wǎng)絡(luò)共享底層資源。與傳統(tǒng)計(jì)算模式不同,云計(jì)算通過虛擬化技術(shù)實(shí)現(xiàn)對硬件資源的虛擬化管理、調(diào)度與應(yīng)用,軟件／硬件組件異構(gòu)以及它們之間復(fù)雜的相互關(guān)系增加了底層資源調(diào)度的難度,使得資源調(diào)度成為云計(jì)算研究的重要方面。 云計(jì)算中的資源調(diào)度主要包括數(shù)據(jù)資源的可靠存儲以及用戶任務(wù)的有效分配兩方面。在可靠性方面,當(dāng)前云計(jì)算技術(shù)大多通過數(shù)據(jù)復(fù)制實(shí)現(xiàn)在廉價(jià)服務(wù)器集群上構(gòu)建穩(wěn)定可靠的分布式文件系統(tǒng),如何對云服務(wù)可靠性進(jìn)行評估成為云計(jì)算可靠性分析的重要方面。在有效性方面,對任務(wù)進(jìn)行合理的分配,可以有效提高系統(tǒng)整體的性能。為此,本文對云計(jì)算的資源調(diào)度進(jìn)行了深入分析,并在此基礎(chǔ)上實(shí)現(xiàn)理論創(chuàng)新和實(shí)驗(yàn)驗(yàn)證。本文的創(chuàng)新點(diǎn)如下： (1)本文在傳統(tǒng)可靠性分析方法的基礎(chǔ)上提出兩種低復(fù)雜度的可靠性評估算法,分別對故障相互獨(dú)立和故障相互關(guān)聯(lián)兩種情況下的云服務(wù)可靠性進(jìn)行評估,前者通過邊界方法對計(jì)算方法進(jìn)行簡化,在保證可靠性精度的前提下降低了計(jì)算的難度,后者利用貝葉斯網(wǎng)絡(luò)和馬爾科夫理論對故障關(guān)聯(lián)性進(jìn)行模擬,并提出一種簡單的可靠性計(jì)算算法。 (2)本文提出基于混沌蟻群算法的云計(jì)算任務(wù)調(diào)度策略,以便解決異構(gòu)環(huán)境下云計(jì)算任務(wù)的分配。為了保證用戶服務(wù)質(zhì)量,我們對任務(wù)的完成時(shí)間、可靠性等諸多方面進(jìn)行了分析,并在此基礎(chǔ)上構(gòu)建了帶有約束條件的多目標(biāo)調(diào)度模型。本文利用混沌蟻群算法對云計(jì)算調(diào)度問題進(jìn)行求解,實(shí)驗(yàn)結(jié)果表明混沌蟻群算法可以有效提高用戶的服務(wù)質(zhì)量,且性能優(yōu)于其他群體智能算法。 (3) Hadoop作為當(dāng)前云計(jì)算中的主流技術(shù),構(gòu)建在分布式文件系統(tǒng)HDFS上,采用MapReduce編程模型處理任務(wù)。為了具體了解云計(jì)算的運(yùn)轉(zhuǎn)模式和任務(wù)調(diào)度流程,本文對Hadoop中的關(guān)鍵技術(shù)進(jìn)行了深入的剖析,尤其是對常用的容量調(diào)度器、公平調(diào)度器等調(diào)度機(jī)制進(jìn)行了分析和闡述。 (4)實(shí)驗(yàn)室搭建了基于HP服務(wù)器的Hadoop云計(jì)算平臺,并在平臺上對聯(lián)通用戶上網(wǎng)數(shù)據(jù)開展分析和統(tǒng)計(jì)工作,完成用戶分類、流量預(yù)測、流向分析及網(wǎng)頁關(guān)鍵詞提取等功能,從而對Hadoop處理流程和任務(wù)調(diào)度機(jī)制有了充分的認(rèn)識。 (5)本文提出了基于資源感知的Hadoop任務(wù)調(diào)度機(jī)制,它通過對底層資源的監(jiān)測獲取節(jié)點(diǎn)的資源使用狀況,從而為任務(wù)的調(diào)度提供參考。另外,在作業(yè)的調(diào)度方面,本文提出基于剩余時(shí)間預(yù)測的作業(yè)選擇策略,通過對作業(yè)剩余運(yùn)行時(shí)間的估算對作業(yè)進(jìn)行排序,并優(yōu)先調(diào)度剩余時(shí)間較短的作業(yè),可以在一定程度上增加系統(tǒng)數(shù)據(jù)處理的時(shí)效性。
[Abstract]:With the increasing demand for computing power, the development of high-performance computing technology spawned the concept of cloud computing. As a new service and computing model, cloud computing makes computing power and storage resources available on demand, and user services and applications share the underlying resources through the network. Different from the traditional computing mode, cloud computing implements virtualization management, scheduling and application of hardware resources, heterogeneous software / hardware components and complex relationships between them through virtualization technology, which increases the difficulty of resource scheduling. Resource scheduling has become an important aspect of cloud computing research. Resource scheduling in cloud computing mainly includes two aspects: reliable storage of data resources and efficient assignment of user tasks. In terms of reliability, most of cloud computing technologies build stable and reliable distributed file systems on cheap server clusters through data replication. How to evaluate the reliability of cloud services becomes an important aspect of cloud computing reliability analysis. In terms of effectiveness, a reasonable assignment of tasks can effectively improve the overall performance of the system. In this paper, the resource scheduling of cloud computing is deeply analyzed, and theoretical innovation and experimental verification are realized on this basis. The innovations of this paper are as follows: In this paper, based on the traditional reliability analysis methods, two low complexity reliability evaluation algorithms are proposed to evaluate the cloud service reliability under the condition of fault mutual independence and fault correlation respectively. The former simplifies the calculation method by boundary method and reduces the difficulty of calculation on the premise of ensuring reliability accuracy. The latter uses Bayesian network and Markov theory to simulate the fault correlation. A simple reliability calculation algorithm is proposed. This paper proposes a cloud computing task scheduling strategy based on chaotic ant colony algorithm to solve the problem of cloud computing task allocation in heterogeneous environment. In order to ensure the quality of service (QoS) of users, we analyze the completion time and reliability of the task, and build a multi-objective scheduling model with constraints. In this paper, chaotic ant colony algorithm is used to solve cloud computing scheduling problem. Experimental results show that chaotic ant colony algorithm can effectively improve the quality of service of users, and its performance is better than other swarm intelligence algorithms. As the mainstream technology of cloud computing, Hadoop is built on the distributed file system (HDFS), and uses MapReduce programming model to deal with the task. In order to understand the operation mode and task scheduling process of cloud computing, this paper analyzes the key technologies in Hadoop, especially the common scheduling mechanisms such as capacity scheduler and fair scheduler. The Hadoop cloud computing platform based on HP server has been set up in the laboratory. On the platform, the data of Unicom users are analyzed and counted, and the functions of user classification, traffic prediction, flow analysis and page keyword extraction are completed. Therefore, the Hadoop processing flow and task scheduling mechanism are fully understood. In this paper, a resource-aware Hadoop task scheduling mechanism is proposed, which obtains the resource usage of the node by monitoring the underlying resources, thus providing a reference for task scheduling. In addition, in the aspect of job scheduling, this paper proposes a job selection strategy based on the prediction of residual time, and gives priority to scheduling jobs with shorter residual time by estimating the remaining running time of jobs. It can increase the timeliness of system data processing to a certain extent.
【學(xué)位授予單位】：北京郵電大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2014
【分類號】：TP393.09;TP18

【參考文獻(xiàn)】

相關(guān)期刊論文 前1條

1 劉萬軍;張孟華;郭文越;;基于MPSO算法的云計(jì)算資源調(diào)度策略[J];計(jì)算機(jī)工程;2011年11期

，

本文編號：1905188