本文選題：軍事信息系統(tǒng) + 一體化信息基礎(chǔ)設(shè)施�。� 參考：《國(guó)防科學(xué)技術(shù)大學(xué)》2012年博士論文

【摘要】：從底層的硬件架構(gòu)到上層的數(shù)據(jù)處理方式，數(shù)據(jù)中心網(wǎng)絡(luò)和MapReduce共同構(gòu)成數(shù)據(jù)中心與云計(jì)算的核心體系，同時(shí)也成為一體化信息基礎(chǔ)設(shè)施快速處理大量軍情信息和數(shù)據(jù)的關(guān)鍵技術(shù)，是未來(lái)網(wǎng)絡(luò)中心戰(zhàn)中奪取制信息權(quán)的必備條件。近年來(lái)，各種新的服務(wù)需求的不斷涌現(xiàn)對(duì)數(shù)據(jù)中心網(wǎng)絡(luò)提出了更高的結(jié)構(gòu)性要求。針對(duì)這些要求，研究者們?cè)O(shè)計(jì)了一些新的數(shù)據(jù)中心網(wǎng)絡(luò)結(jié)構(gòu)。其中遞歸層次結(jié)構(gòu)相對(duì)其它結(jié)構(gòu)，具有更強(qiáng)的自組織性、更高的可靠性、更好的可拓展性等優(yōu)點(diǎn)，在軍用和民用領(lǐng)域都具有重要研究?jī)r(jià)值和廣泛應(yīng)用前景。但研究者們僅僅從結(jié)構(gòu)本身來(lái)考慮如何提高網(wǎng)絡(luò)性能，卻忽略了數(shù)據(jù)中心網(wǎng)絡(luò)設(shè)計(jì)的實(shí)用性要求，特別是與MapReduce的數(shù)據(jù)處理機(jī)制相適應(yīng)的問(wèn)題。本文針對(duì)一體化信息基礎(chǔ)設(shè)施中遞歸層次結(jié)構(gòu)數(shù)據(jù)中心網(wǎng)絡(luò)與MapReduce的匹配這一議題展開(kāi)研究。 主要研究工作和創(chuàng)新點(diǎn)如下： 1）提出了遞歸層次結(jié)構(gòu)數(shù)據(jù)中心網(wǎng)絡(luò)的可靠性分析方法 出于軍事需求考慮，提出了分析判斷遞歸層次結(jié)構(gòu)數(shù)據(jù)中心網(wǎng)絡(luò)可靠性的系統(tǒng)方法。從拓?fù)湓O(shè)計(jì)的角度分析了遞歸層次結(jié)構(gòu)數(shù)據(jù)中心網(wǎng)絡(luò)可靠性的評(píng)價(jià)指標(biāo)，包括連通性、聚合性、以及敏感性�；谶f歸層次結(jié)構(gòu)數(shù)據(jù)中心網(wǎng)絡(luò)的形式化描述，針對(duì)每個(gè)可靠性評(píng)價(jià)指標(biāo)，研究了具體的量化評(píng)價(jià)分析方法。利用DCell、FiConn和BCube這三種目前最為典型的遞歸層次結(jié)構(gòu)為案例，檢驗(yàn)了方法的可行性和有效性。通過(guò)案例分析發(fā)現(xiàn)，雖然FiConn的敏感性最好，但其聚合性和連通性最差，綜合三個(gè)評(píng)價(jià)指標(biāo)的分析結(jié)果，得出BCube的可靠性最高。 2）提出了MapReduce程序設(shè)計(jì)的合理性分析方法 基于對(duì)象Petri網(wǎng)，提出了一種全面分析驗(yàn)證MapReduce程序設(shè)計(jì)合理性的系統(tǒng)方法�？偨Y(jié)出分析MapReduce程序設(shè)計(jì)合理性的具體目的，并基于這些目的研究給出MapReduce程序的合理性指標(biāo)，包括具有邏輯上可執(zhí)行的工作流程、不存在Straggler和Map沖突、具有合理的運(yùn)行時(shí)間，以此判斷MapReduce程序是否存在設(shè)計(jì)不合理問(wèn)題。由于對(duì)象Petri網(wǎng)能夠很好地描述復(fù)雜MapReduce程序的內(nèi)在關(guān)系，準(zhǔn)確無(wú)誤的模擬MapReduce程序中各個(gè)步驟的執(zhí)行情況，并且在模擬過(guò)程中不需要人工干預(yù)，因此利用對(duì)象Petri網(wǎng)模擬MapReduce的數(shù)據(jù)處理步驟。針對(duì)各個(gè)合理性指標(biāo)采取不同的方法分析對(duì)象Petri網(wǎng)的運(yùn)行過(guò)程和結(jié)果判斷MapReduce程序是否存在設(shè)計(jì)不合理的問(wèn)題，以達(dá)到不通過(guò)在數(shù)據(jù)中心網(wǎng)絡(luò)上運(yùn)行MapReduce程序就能驗(yàn)證其合理性的目的。通過(guò)針對(duì)每條合理性指標(biāo)的實(shí)驗(yàn)分析和結(jié)果，，證明了方法的有效性。 3）設(shè)計(jì)了一種支持MapReduce的遞歸層次結(jié)構(gòu) 基于BCube和Fat-tree結(jié)構(gòu)，設(shè)計(jì)了一種支持MapReduce的遞歸層次結(jié)構(gòu)——Hyper-Fat-tree Network（HFN）。HFN的構(gòu)建方法基于BCube的遞歸規(guī)律，即以一個(gè)低層的網(wǎng)絡(luò)拓?fù)渥鳛橐粋�(gè)遞歸單元，多個(gè)這樣的遞歸單元按照超立方體（hypercube）的節(jié)點(diǎn)連接關(guān)系構(gòu)成一個(gè)高一層的網(wǎng)絡(luò)拓?fù)洹５煌贐Cube的是，HFN的最小遞歸單元采用類(lèi)似Fat-tree的冗余構(gòu)造，并依據(jù)MapReduce的執(zhí)行控制過(guò)程，明確了主服務(wù)器和工作服務(wù)器的相對(duì)位置和連接關(guān)系，以適應(yīng)MapReduce的數(shù)據(jù)處理機(jī)制和提高利用MapReduce進(jìn)行分布式數(shù)據(jù)處理的可靠性。由于結(jié)合了超立方體和Fat-tree的優(yōu)點(diǎn)，HFN具有連通性高、直徑小、可靠性好的特點(diǎn)。HFN還具有較好的可拓展性，HFN可連接的服務(wù)器數(shù)量是BCube中服務(wù)器數(shù)量的數(shù)倍，有效滿(mǎn)足了一體化信息基礎(chǔ)設(shè)施建設(shè)發(fā)展對(duì)數(shù)據(jù)中心網(wǎng)絡(luò)服務(wù)器數(shù)量不斷增長(zhǎng)的要求。 4）提出了在一般遞歸層次結(jié)構(gòu)數(shù)據(jù)中心網(wǎng)絡(luò)上組織維護(hù)數(shù)據(jù)文件的方法 基于分布式哈希表的基本原理，提出了一種在一般遞歸層次結(jié)構(gòu)數(shù)據(jù)中心網(wǎng)絡(luò)上組織維護(hù)數(shù)據(jù)文件的方法。針對(duì)遞歸層次結(jié)構(gòu)中服務(wù)器的互聯(lián)方式，確定了服務(wù)器在組織維護(hù)數(shù)據(jù)文件過(guò)程中的作用。利用分布式哈希表的基本原理，研究了在一般遞歸層次結(jié)構(gòu)數(shù)據(jù)中心網(wǎng)絡(luò)上實(shí)現(xiàn)數(shù)據(jù)存儲(chǔ)、讀取和維護(hù)的方法。介紹了這些方法中涉及的分布式哈希表的鍵表結(jié)構(gòu)，以及基于此結(jié)構(gòu)進(jìn)行數(shù)據(jù)文件維護(hù)操作的路由方法。針對(duì)服務(wù)器故障，給出了在一般遞歸層次結(jié)構(gòu)數(shù)據(jù)中心網(wǎng)絡(luò)上組織和維護(hù)數(shù)據(jù)文件的容錯(cuò)方法。通過(guò)實(shí)驗(yàn)分析比較了在HFN和BCube上進(jìn)行數(shù)據(jù)操作的平均路徑長(zhǎng)度和考慮節(jié)點(diǎn)故障時(shí)數(shù)據(jù)操作的成功率，證明了此方法能夠?qū)⒏鞣N數(shù)據(jù)操作請(qǐng)求信息迅速有效地發(fā)送給相應(yīng)的目的服務(wù)器，利用此方法在一般遞歸層次結(jié)構(gòu)數(shù)據(jù)中心網(wǎng)絡(luò)上組織和維護(hù)數(shù)據(jù)文件是可行且有效的。 5）提出了在一般遞歸層次結(jié)構(gòu)數(shù)據(jù)中心網(wǎng)絡(luò)上運(yùn)行MapReduce的方法 基于分布式哈希表的基本原理，提出了一種在一般遞歸層次結(jié)構(gòu)數(shù)據(jù)中心網(wǎng)絡(luò)上運(yùn)行MapReduce的方法。針對(duì)MapReduce的基本數(shù)據(jù)處理機(jī)制，研究了在一般遞歸層次結(jié)構(gòu)數(shù)據(jù)中心網(wǎng)絡(luò)上如何區(qū)分主服務(wù)器和工作服務(wù)器，以及分配Map和Reduce任務(wù)、傳輸中間數(shù)據(jù)的方法。介紹了這些方法中涉及的分布式哈希表的鍵表結(jié)構(gòu)，以及基于此結(jié)構(gòu)進(jìn)行任務(wù)分配和中間數(shù)據(jù)傳輸?shù)穆酚煞椒�。研究了MapReduce在一般遞歸層次結(jié)構(gòu)數(shù)據(jù)中心網(wǎng)絡(luò)上的容錯(cuò)方法，包括容錯(cuò)路由以及針對(duì)服務(wù)器故障的容錯(cuò)方法。通過(guò)實(shí)驗(yàn)分析比較了HFN與BCube在利用此方法運(yùn)行MapReduce時(shí)的網(wǎng)絡(luò)性能，包括負(fù)載平衡、吞吐量和帶寬，證明了此方法能夠?qū)⒄麄�(gè)數(shù)據(jù)中心網(wǎng)絡(luò)的負(fù)載平均分配給各臺(tái)服務(wù)器，并且在節(jié)點(diǎn)故障率較高的情況下依然能夠滿(mǎn)足帶寬需求量大的MapReduce應(yīng)用和服務(wù)。
[Abstract]:From the underlying hardware architecture to the data processing on the upper level, the data center network and the MapReduce constitute the core of the data center and cloud computing. At the same time, it is also the key technology for the integration of information infrastructure to quickly deal with a large number of military information and data. It is a necessary condition for the acquisition of information right in the future network center war. Over the years, the growing demand for new services has made a higher structural requirement for the data center network. In response to these requirements, researchers have designed some new data center network structures, in which the recursive hierarchy is more self-organized, more reliable, and more expansibility, and so on. It has important research value and wide application prospects in both military and civil fields. But researchers only consider how to improve the network performance from the structure itself, but ignore the practical requirements of the data center network design, especially the problem that adapts to the data processing mechanism of MapReduce. This paper aims at the integrated information infrastructure. The issue of the matching between the recursive hierarchical data center network and MapReduce is studied.
The main research work and innovation are as follows:
1) the reliability analysis method of recursive hierarchical data center network is proposed.
A systematic method of analyzing and judging the reliability of the recurrent hierarchical structure data center network is proposed for the consideration of military needs. The evaluation indexes of the reliability of the recursive hierarchical structure data center network are analyzed from the point of view of topology design, including connectivity, aggregation, and sensitivity. The feasibility and effectiveness of the three most typical recursive hierarchical structures of DCell, FiConn and BCube are examined. The case analysis shows that although the sensitivity of FiConn is the best, its aggregation and connectivity is the worst, Combining the analysis results of three evaluation indexes, it is concluded that the reliability of BCube is the highest.
2) put forward the rationality analysis method of MapReduce programming.
Based on object Petri net, this paper presents a systematic method to comprehensively analyze and verify the rationality of MapReduce program design, and summarizes the specific purpose of analyzing the rationality of MapReduce program design. Based on these purposes, the rationality index of the MapReduce program is given, including the logically executable workflow, and there is no Straggler and Map The conflict, with a reasonable running time, can judge whether the MapReduce program has the problem of unreasonable design. Because the object Petri network can describe the internal relationship of the complex MapReduce program well, accurately simulate the execution of each step in the MapReduce program, and do not need manual intervention in the simulation process. The object Petri network simulates the data processing steps of the MapReduce. According to each reasonable index, it takes different methods to analyze the operation process and the result of the object Petri net to judge whether the MapReduce program has the unreasonable design problem, so as to achieve the purpose of verifying its rationality without running the MapReduce program on the data center network. The effectiveness and effectiveness of the method are proved by analyzing the results of each reasonable index.
3) a recursive hierarchical structure supporting MapReduce is designed.
Based on the structure of BCube and Fat-tree, a recursive hierarchical structure supporting MapReduce, Hyper-Fat-tree Network (HFN).HFN, is designed based on the recursive law of BCube, that is, a low-level network topology is used as a recursive unit, and many such recursion units are connected by the nodes of the hypercube (hypercube). It is a high level network topology. But unlike BCube, the minimum recursive unit of HFN uses redundant constructs similar to Fat-tree. According to the execution control process of MapReduce, the relative position and connection relationship between the main server and the work server is defined to adapt to the MapReduce data processing mechanism and improve the use of MapReduce. The reliability of distributed data processing. Because of the advantages of hypercube and Fat-tree, HFN has high connectivity, small diameter, and good reliability,.HFN also has good expansibility. The number of HFN connected servers is several times the number of servers in BCube, and it is full of integrated information infrastructure development to data. The demand for the growing number of central network servers.
4) a method of organizing and maintaining data files on a general recursive hierarchical data center network is proposed.
Based on the basic principle of distributed hash table, a method of organizing and maintaining data files on a general recursive hierarchical structure data center network is proposed. The application of the server in the process of organizing and maintaining data files is determined. The basic principle of the distributed hash table is studied. The method of data storage, reading and maintenance on the general recursive hierarchical data center network is introduced. The key table structure of the distributed hash table involved in these methods, and the routing method based on this structure for data file maintenance operation are introduced. The general recursive hierarchical structure data center is given for the server failure. The fault tolerance method of organizing and maintaining data files on the network. The average path length of data operation on HFN and BCube and the success rate of data operation considering the node failure are compared by experiment analysis. It is proved that this method can send various data operation request information to the corresponding destination server quickly and effectively, and use this method to use this method. It is feasible and effective to organize and maintain data files on general recursive hierarchical data center networks.
5) a method of running MapReduce on general recursive hierarchical data center network is proposed.
Based on the basic principle of distributed hash table, a method of running MapReduce on a general recursive hierarchical data center network is proposed. Based on the basic data processing mechanism of MapReduce, the paper studies how to distinguish between the master server and the work server on the general recursive hierarchical data center network and the allocation of Map and Reduce. The method of transferring intermediate data. This paper introduces the key table structure of distributed hash table involved in these methods, and the routing method based on this structure for task assignment and intermediate data transmission. It studies the fault tolerant methods of MapReduce in the general recursive hierarchical data center network, including fault-tolerant routing and for the server. Through experimental analysis, the network performance of HFN and BCube when using this method to run MapReduce, including load balance, throughput and bandwidth, proves that this method can allocate the load of the whole data center network to each server, and can still be satisfied when the node failure rate is high. MapReduce applications and services with large bandwidth requirements.
【學(xué)位授予單位】：國(guó)防科學(xué)技術(shù)大學(xué)
【學(xué)位級(jí)別】：博士
【學(xué)位授予年份】：2012
【分類(lèi)號(hào)】：TP308

【參考文獻(xiàn)】

相關(guān)期刊論文 前4條

1 李慧波;王源;;一體化信息基礎(chǔ)設(shè)施環(huán)境下的WebGIS研究[J];兵工自動(dòng)化;2011年02期

2 劉俊先,羅雪山;對(duì)象Petri網(wǎng)及其在C~4ISR系統(tǒng)仿真中的應(yīng)用[J];計(jì)算機(jī)仿真;2003年03期

3 張路青;;海上一體化信息基礎(chǔ)設(shè)施總體技術(shù)框架研究[J];艦船電子工程;2010年03期

4 張培珍;楊根源;平殿發(fā);武志東;王煥章;;基于GIG應(yīng)用的服務(wù)質(zhì)量體系結(jié)構(gòu)研究[J];計(jì)算機(jī)測(cè)量與控制;2010年04期

本文編號(hào)：1956872