本文選題：大規(guī)模圖數(shù)據(jù)處理 + Pregel系統(tǒng)��； 參考：《東南大學(xué)》2016年碩士論文

【摘要】：隨著電子商務(wù)、移動(dòng)互聯(lián)網(wǎng)、物聯(lián)網(wǎng)等技術(shù)的進(jìn)一步發(fā)展,數(shù)據(jù)的規(guī)模、產(chǎn)生速度、復(fù)雜性均日益增長(zhǎng),標(biāo)志著人類社會(huì)已經(jīng)進(jìn)入了大數(shù)據(jù)時(shí)代。隨著數(shù)據(jù)之間的聯(lián)系變得更加緊密、依賴關(guān)系更加復(fù)雜,部分?jǐn)?shù)據(jù)的分布模式逐漸具有圖的特征。傳統(tǒng)的大數(shù)據(jù)處理技術(shù),如MapReduce批處理框架不適用于關(guān)系復(fù)雜、需要多次迭代的圖數(shù)據(jù)。谷歌的Pregel系統(tǒng)通過(guò)并行化的思想,并行地進(jìn)行頂點(diǎn)計(jì)算,大大提高了計(jì)算性能,為大規(guī)模圖數(shù)據(jù)處理提供了新的思路。現(xiàn)有大規(guī)模圖數(shù)據(jù)處理的研究工作均基于Pregel的思想,部分解決了大規(guī)模圖數(shù)據(jù)處理的問(wèn)題,但仍存在如下的問(wèn)題：一方面忽略了未隔離應(yīng)用間的資源競(jìng)爭(zhēng)而造成的性能衰減；另一方面則忽略了應(yīng)用各階段對(duì)資源的彈性需求而造成的性能下降或者資源浪費(fèi)問(wèn)題。為了解決以上的問(wèn)題,本碩士論文將虛擬化技術(shù)引入圖數(shù)據(jù)處理中,通過(guò)深入分析圖數(shù)據(jù)處理過(guò)程的執(zhí)行特性,并結(jié)合虛擬化良好的進(jìn)程隔離性以及靈活的資源彈性可管理性,提出面向應(yīng)用的圖劃分以及資源分配與調(diào)度機(jī)制,實(shí)現(xiàn)根據(jù)應(yīng)用的具體執(zhí)行模式來(lái)進(jìn)行資源的彈性供給,以提高圖數(shù)據(jù)處理系統(tǒng)的整體執(zhí)行效率。綜上所述,本論文從以下四個(gè)方面開展研究工作：首先,研究大規(guī)模圖數(shù)據(jù)處理應(yīng)用執(zhí)行模式的抽取和分析機(jī)制�；陂_源類Pregel系統(tǒng)進(jìn)行二次開發(fā),完成對(duì)應(yīng)用執(zhí)行模式的抽取,并建立執(zhí)行模式與底層資源需求間的映射關(guān)系,為后續(xù)虛擬資源的分配與調(diào)度提供可靠的理論依據(jù),是本文研究工作的基礎(chǔ)。其次,研究應(yīng)用感知的大規(guī)模圖數(shù)據(jù)劃分方法。大規(guī)模圖數(shù)據(jù)劃分是并行計(jì)算的前提。本文在虛擬化環(huán)境中,根據(jù)應(yīng)用的執(zhí)行模式,對(duì)圖數(shù)據(jù)進(jìn)行合理地劃分,從而減少網(wǎng)絡(luò)通訊并且實(shí)現(xiàn)負(fù)載均衡。對(duì)圖數(shù)據(jù)進(jìn)行合理地劃分一方面有助于更好地進(jìn)行資源分配與調(diào)度,另一方面可以提升應(yīng)用的執(zhí)行性能。再次,研究面向應(yīng)用執(zhí)行模式的虛擬資源分配與調(diào)度機(jī)制�；趫�(zhí)行模式與底層資源需求的映射關(guān)系,設(shè)計(jì)面向應(yīng)用的虛擬資源分配與調(diào)度機(jī)制,根據(jù)上層應(yīng)用執(zhí)行模式對(duì)資源進(jìn)行細(xì)粒度的分配與調(diào)度,在保證上層應(yīng)用性能的前提下提高資源利用率。最后,通過(guò)部署Openstack軟件以實(shí)現(xiàn)虛擬化環(huán)境,在此基礎(chǔ)上,實(shí)現(xiàn)了相關(guān)理論的研究工作,設(shè)計(jì)并開發(fā)了大規(guī)模圖數(shù)據(jù)處理平臺(tái)nutcat以集成應(yīng)用特征抽取模塊,應(yīng)用感知的超塊劃分模塊以及面向應(yīng)用執(zhí)行模式的虛擬資源分配與調(diào)度模塊,并部署于東南大學(xué)云計(jì)算中心(SEU CLOUD)真實(shí)環(huán)境中。通過(guò)在真實(shí)的東南大學(xué)云計(jì)算中心環(huán)境中的實(shí)驗(yàn)結(jié)果表明,本文提出的應(yīng)用感知的大規(guī)模圖數(shù)據(jù)劃分方法以及面向應(yīng)用執(zhí)行模式的虛擬資源分配與調(diào)度機(jī)制可以顯著地提高應(yīng)用的執(zhí)行性能并提升虛擬資源利用率,并為大數(shù)據(jù)應(yīng)用與虛擬化環(huán)境相結(jié)合提供了面向應(yīng)用進(jìn)行資源分配與調(diào)度的新思路。
[Abstract]:With the further development of e-commerce, mobile Internet, Internet of things and other technologies, the scale, speed and complexity of the data are increasing, which indicates that the human society has entered the era of big data. With the connection of data become more closely, the dependence relationship is more complex, and the distribution pattern of some data has the characteristics of the graph gradually. The traditional large data processing technology, such as the MapReduce batch processing framework, is not suitable for complex relations and needs multiple iterations of graph data. Google's Pregel system performs vertex computation in parallel through parallel thinking, which greatly improves the computing performance and provides a new way of thinking for large scale graph data processing. The research work is based on the idea of Pregel, which partly solves the problem of data processing in large scale maps, but there are still some problems as follows: on the one hand, it ignores the performance attenuation caused by the resource competition between non isolated applications, and on the other hand, it neglects the performance degradation or resource wave caused by the elastic demand of resources at various stages of application. In order to solve the above problems, this thesis introduces the virtualization technology into the graph data processing, analyzes the execution characteristics of the data processing process, and proposes the application oriented graph division and resource allocation and scheduling mechanism by combining the good process isolation of the virtualization process and flexible resource flexibility manageability. To implement the flexible supply of resources according to the specific implementation mode of the application to improve the overall execution efficiency of the map data processing system. In summary, this thesis will carry out the research work from the following four aspects: first, study the extraction and analysis mechanism of the application execution mode of large-scale map data processing. Based on the open source Pregel system The two development, completing the extraction of application execution pattern, and establishing the mapping relationship between the execution mode and the underlying resource requirements, providing a reliable theoretical basis for the distribution and scheduling of the subsequent virtual resources, is the foundation of the research work. Secondly, it studies the method of data partition of the large-scale pattern map of the application perception. In this paper, in the virtualization environment, according to the implementation mode of the application, the graph data is divided reasonably, thus reducing the network communication and realizing the load balancing. The rational partition of the graph data is helpful to the better allocation and scheduling of resources, on the other hand, the performance of the application can be improved. Again, Research on the virtual resource allocation and scheduling mechanism oriented to application implementation mode. Based on the mapping relationship between the execution mode and the underlying resource requirements, the application oriented allocation and scheduling mechanism of virtual resources is designed. According to the upper application execution mode, the resource is distributed and scheduling, and the performance of the upper application is guaranteed. In the end, by deploying Openstack software to realize the virtualization environment, on this basis, the research work of related theories is realized, and a large scale map data processing platform nutcat is designed and developed to integrate application feature extraction modules, use perceived block partition block and Virtual Resource Allocation Oriented to application execution mode. With the scheduling module, and deployed in the real environment of the SEU CLOUD center of Southeast University, the experimental results in the real environment of the Southeast University Cloud Computing Center show that the proposed application aware mass graph data partition method and the application execution mode oriented virtual resource allocation and scheduling mechanism can be significant It improves the performance of the application and improves the utilization of virtual resources, and provides a new idea for the application of resource allocation and scheduling for the combination of large data application and virtual environment.
【學(xué)位授予單位】：東南大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2016
【分類號(hào)】：TP311.52;TP302

【相似文獻(xiàn)】

相關(guān)期刊論文 前1條

1 黃顯凱;;EuP導(dǎo)入建議與生態(tài)化設(shè)計(jì)符合性執(zhí)行模式[J];電子與電腦;2008年07期

相關(guān)重要報(bào)紙文章 前10條

1 本報(bào)記者 舒沁　本報(bào)通訊員 余寧;分段集約機(jī)制 優(yōu)化執(zhí)行模式[N];人民法院報(bào);2011年

2 牡丹江市委市政府督查辦暨“三實(shí)兩創(chuàng)”辦公室;探索城市的執(zhí)行模式[N];學(xué)習(xí)時(shí)報(bào);2013年

3 王國(guó)軍　記者 姜文明;創(chuàng)新執(zhí)行模式 破解執(zhí)行難題[N];北方法制報(bào);2009年

4 北京市西城區(qū)人民法院 張緯;“分段集約”誠(chéng)可貴 傳統(tǒng)優(yōu)勢(shì)不可拋[N];人民法院報(bào);2012年

5 記者 婁銀生;徐州“泉山執(zhí)行模式”帶動(dòng)各地破解司法難題[N];人民法院報(bào);2012年

6 記者 謝曉曦　通訊員 張瑞雪;重慶打破包案到底執(zhí)行模式[N];人民法院報(bào);2009年

7 倪志祥;傳統(tǒng)執(zhí)行模式存在弊端[N];江蘇經(jīng)濟(jì)報(bào);2013年

8 特約通訊員 曉梅　秋蘇;出彩的“全員能動(dòng)執(zhí)行模式”[N];徐州日?qǐng)?bào);2011年

9 吳歡 朱益虎;常熟多維度創(chuàng)新執(zhí)行模式提升執(zhí)法效果[N];江蘇經(jīng)濟(jì)報(bào);2014年

10 江西省石城縣人民法院 陳默;“泉山模式”以能動(dòng)司法破解司法難題[N];人民法院報(bào);2012年

相關(guān)碩士學(xué)位論文 前5條

1 徐海茹;盲量子計(jì)算協(xié)議與執(zhí)行模式研究[D];廣東工業(yè)大學(xué);2016年

2 李小龍;廣州市新能源公交車推廣政策執(zhí)行模式分析及效果評(píng)價(jià)研究[D];華南農(nóng)業(yè)大學(xué);2016年

3 張駿雪;面向大規(guī)模圖數(shù)據(jù)處理的虛擬機(jī)管理系統(tǒng)研究與實(shí)現(xiàn)[D];東南大學(xué);2016年

4 王子睿;非訴行政執(zhí)行模式研究[D];西南政法大學(xué);2015年

5 陳靜;生產(chǎn)性服務(wù)業(yè)市場(chǎng)導(dǎo)向執(zhí)行模式與績(jī)效相關(guān)性研究[D];沈陽(yáng)工業(yè)大學(xué);2012年

，

本文編號(hào)：1898539