本文關(guān)鍵詞： 智能電網(wǎng)云平臺 Hadoop 作業(yè)調(diào)度 云計算 推測執(zhí)行任務(wù)　出處：《華北電力大學(xué)》2014年碩士論文　論文類型：學(xué)位論文

【摘要】：隨著我國智能電網(wǎng)事業(yè)的發(fā)展，全國電力系統(tǒng)互聯(lián)已成為一個趨勢，大量的先進(jìn)的數(shù)據(jù)采集與監(jiān)控設(shè)備、相量測量單元（PMU）、智能電表等被應(yīng)用，現(xiàn)代電力系統(tǒng)正在演變成一個集聚大數(shù)據(jù)和信息的計算系統(tǒng)。針對智能電網(wǎng)對海量的數(shù)據(jù)存儲和大規(guī)模并行計算的迫切需求，鑒于電力系統(tǒng)廣域網(wǎng)的完整性，學(xué)者提出了整合網(wǎng)內(nèi)現(xiàn)有計算和存儲資源，建立電力私有云的概念。Hadoop是主要由HDFS和MapReduce組成的開源云計算項目，可以部署在普通個人計算機(jī)上，從而組成廉價的云平臺。作業(yè)的調(diào)度算法對云計算有著至關(guān)重要的作用，它是解決作業(yè)在什么地點、什么時間執(zhí)行的問題。智能電網(wǎng)云平臺依托于各級電網(wǎng)的計算資源，集群中普遍資源存在著節(jié)點異構(gòu)問題，異構(gòu)節(jié)點的執(zhí)行能力的不同和用戶提交作業(yè)任務(wù)量不同，會導(dǎo)致比較突出的任務(wù)同步問題。 根據(jù)該情況，本文在hadoop平臺下，給出了一種基于作業(yè)執(zhí)行時間預(yù)測的資源優(yōu)化推測執(zhí)行算法，該算法通過預(yù)先執(zhí)行作業(yè)一部分任務(wù)，通過這些先行任務(wù)預(yù)測作業(yè)平均和整體的運行時間，同時將群集中的節(jié)點以執(zhí)行相同作業(yè)所屬任務(wù)的執(zhí)行時間為參數(shù)，將節(jié)點分為快節(jié)點和慢節(jié)點，而推測執(zhí)行的任務(wù)只能發(fā)生在快節(jié)點上，該算法結(jié)合任務(wù)執(zhí)行節(jié)點的性能參數(shù)，判斷該任務(wù)是否進(jìn)行推測執(zhí)行，當(dāng)推測執(zhí)行發(fā)生時會盡可能以局部執(zhí)行的方式執(zhí)行其后備任務(wù)，推測任務(wù)發(fā)生之前，，該算法會檢查群集中其它節(jié)點執(zhí)行該任務(wù)的成本是否低于該節(jié)點（主要以inputsplit的所在節(jié)點與執(zhí)行節(jié)點的距離做參考），如果任務(wù)在其它節(jié)點執(zhí)行成本更低，則算法會放棄本次推測執(zhí)行。本文通過實驗比較了該算法和、計算能力調(diào)度算法、公平調(diào)度算法、基于高優(yōu)先級滑動窗口調(diào)度算法的優(yōu)缺點，通過分別代表內(nèi)存、CPU、網(wǎng)絡(luò)等不同類型資源的云計算應(yīng)用例程WordCount、CPUActivity、URLGet，進(jìn)行三組，每組六次實驗的測試，結(jié)果表明該算法在任務(wù)的時間消耗上，推測執(zhí)行的發(fā)生率，網(wǎng)絡(luò)資源的占用率上均有明顯的減小，整體上縮短了資源的消耗，并提高了任務(wù)的完成速度。因此在一定程度上適合節(jié)點眾多，拓?fù)浣Y(jié)構(gòu)復(fù)雜，節(jié)點差異大的電力系統(tǒng)私有云的作業(yè)調(diào)度的需求。
[Abstract]:With the development of smart grid in China, the interconnection of national power system has become a trend. A large number of advanced data acquisition and monitoring equipment, phasor measurement unit (PMU), intelligent meter and so on have been applied. Modern power system is evolving into a computing system that gathers big data and information. In view of the urgent demand of smart grid for massive data storage and large-scale parallel computing, considering the integrity of power system wide area network (WAN). Scholars put forward the concept of integrating the existing computing and storage resources in the network and establishing the power private cloud. Hadoop is an open source cloud computing project mainly composed of HDFS and MapReduce. It can be deployed on an ordinary personal computer to form a cheap cloud platform. Job scheduling algorithms play a vital role in cloud computing, which is a solution to where jobs are located. The problem of when to execute. The cloud platform of smart grid depends on the computing resources of all levels of power grid, and the problem of node heterogeneity exists in the common resources in the cluster. The different execution ability of heterogeneous nodes and the number of jobs submitted by users will lead to the problem of task synchronization. According to this situation, this paper presents a resource optimization algorithm based on job execution time prediction based on hadoop platform, which performs a part of the job tasks in advance. At the same time, the nodes in the cluster are divided into fast node and slow node with the execution time of the same job as the parameter. The proposed task can only occur on the fast node, and the algorithm combines the performance parameters of the task execution node to determine whether the task is supposed to be executed. When speculation execution occurs, it performs its backup tasks as locally as possible, prior to the occurrence of the speculated task. The algorithm checks whether the cost of performing this task for other nodes in the cluster is lower than that of the node (mainly referenced by the distance between the node where the inputsplit is located and the node of execution). If the cost of task execution in other nodes is lower, then the algorithm will give up this speculative execution. This paper compares the algorithm with the computational power scheduling algorithm and the fair scheduling algorithm through experiments. Based on the advantages and disadvantages of high priority sliding window scheduling algorithm, cloud computing application routine WordCount, which represents different types of resources, such as memory, network and so on, is adopted. CPUActivityTurlGet3 groups, each group of six experiments, the results show that the algorithm in the task of time consumption, the incidence of execution. The occupancy rate of network resources is obviously reduced, the overall consumption of resources is shortened, and the speed of task completion is improved. Therefore, to some extent, it is suitable for many nodes and complex topology. The demand for job scheduling of private cloud in power system with large node difference.
【學(xué)位授予單位】：華北電力大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2014
【分類號】：TP393.09;TM73

【相似文獻(xiàn)】

相關(guān)期刊論文 前10條

1 徐一哲;沈瑞寒;;智能電網(wǎng)淺析[J];經(jīng)營管理者;2009年15期

2 傅書

本文編號：1474377