本文關(guān)鍵詞：支持虛擬集群遷移的On-demand文件并行傳輸算法　出處：《吉林大學(xué)》2013年碩士論文　論文類(lèi)型：學(xué)位論文

  更多相關(guān)文章： 虛擬集群 目的端并行 按需調(diào)度 文件傳輸

【摘要】：近年來(lái)，虛擬化技術(shù)與網(wǎng)格技術(shù)廣泛結(jié)合，使得分布式環(huán)境中的虛擬集群常被用來(lái)解決各種并行處理問(wèn)題。虛擬集群的動(dòng)態(tài)性和遷移，使得虛擬集群應(yīng)用經(jīng)常涉及到物理集群間的大文件傳輸問(wèn)題。另外，隨著計(jì)算機(jī)和Internet技術(shù)的廣泛普及，，每天都會(huì)產(chǎn)生海量的數(shù)據(jù)，而且以爆炸式的速度增長(zhǎng)。網(wǎng)格計(jì)算技術(shù)也使得大規(guī)模數(shù)據(jù)密集型應(yīng)用得到廣泛發(fā)展。在這類(lèi)應(yīng)用中，大數(shù)據(jù)的生成集群、存儲(chǔ)集群和處理集群往往分布在相距較遠(yuǎn)的物理位置上。這些數(shù)據(jù)文件從遠(yuǎn)程數(shù)據(jù)采集點(diǎn)匯集到處理中心進(jìn)行計(jì)算、顯示和存儲(chǔ)。由此可知，虛擬集群和數(shù)據(jù)密集型并行處理等問(wèn)題，都需要在廣域網(wǎng)分布式共享計(jì)算環(huán)境中高效地傳輸海量數(shù)據(jù)。在并行處理過(guò)程中，我們希望大數(shù)據(jù)文件能在盡可能短的時(shí)間內(nèi)傳輸?shù)较鄳?yīng)的處理節(jié)點(diǎn)，使得數(shù)據(jù)處理可以并發(fā)進(jìn)行。因此，如何在多集群間快速傳輸大文件，逐漸成為研究的熱點(diǎn)。 目前，許多國(guó)內(nèi)外專(zhuān)家學(xué)者對(duì)并行處理中大文件傳輸算法進(jìn)行了相關(guān)研究，分別從調(diào)度策略和路由策略兩方面，提出一些提高文件傳輸性能的應(yīng)用技術(shù)�，F(xiàn)階段，文件傳輸算法的研究側(cè)重于提高傳輸?shù)牟⑿卸群途W(wǎng)絡(luò)帶寬利用率、縮短批量文件請(qǐng)求的整體傳輸時(shí)間。當(dāng)前工作大多使用多重路徑傳輸、多跳路徑傳輸和多副本等方法實(shí)現(xiàn)文件的并行傳輸，但是沒(méi)有考慮到目的端并行接收文件分片的能力，而只有這樣才能更好地提高傳輸并行度。另外，當(dāng)前的一些研究工作也沒(méi)有考慮到對(duì)批量文件請(qǐng)求在整個(gè)網(wǎng)絡(luò)中傳輸?shù)娜挚刂坪蜎_突協(xié)調(diào)。 本文針對(duì)這類(lèi)傳輸問(wèn)題提出了一個(gè)支持虛擬集群遷移的按需文件并行傳輸算法OFPT（On-demand File Parallel Transfer），OFPT算法的目的是實(shí)現(xiàn)批量文件傳輸請(qǐng)求的整體完成時(shí)間的最小化。該算法根據(jù)集群內(nèi)部數(shù)據(jù)快速傳輸?shù)奶攸c(diǎn)，將目的端擴(kuò)展為集群內(nèi)所有存在外部連接的節(jié)點(diǎn)，實(shí)現(xiàn)目的端并行，分散單個(gè)節(jié)點(diǎn)的傳輸負(fù)載。在傳輸路徑上，采用多重路徑實(shí)現(xiàn)并行傳輸，對(duì)于單一路徑，使用多跳路徑散列的方法并靈活調(diào)整路徑的跳數(shù)限制，以獲取最優(yōu)傳輸路徑。對(duì)于批量文件傳輸請(qǐng)求，依據(jù)每個(gè)請(qǐng)求的傳輸負(fù)載，在全局范圍內(nèi)按需分配網(wǎng)絡(luò)帶寬，解決多個(gè)請(qǐng)求的路徑間的帶寬沖突，從而提高網(wǎng)絡(luò)資源的帶寬利用率，快速實(shí)現(xiàn)傳輸批量文件請(qǐng)求。 本文使用NS2仿真軟件，模擬多個(gè)集群間批量大文件傳輸?shù)膶?shí)驗(yàn)環(huán)境，并進(jìn)行了詳盡的實(shí)驗(yàn)。依次測(cè)試了傳輸模式、文件副本數(shù)、傳輸負(fù)載等因素對(duì)本算法的影響。最終實(shí)驗(yàn)結(jié)果表明，本文提出的按需文件并行傳輸算法OFPT有效地提高了網(wǎng)絡(luò)資源利用率，在吞吐量等傳輸性能上明顯好于當(dāng)前的廣域網(wǎng)中的大文件傳輸算法，達(dá)到了本文工作的預(yù)期。
[Abstract]:In recent years, virtualization technology and grid technology have been widely combined, making the virtual cluster in distributed environment is often used to solve a variety of parallel processing problems, virtual cluster dynamic and migration. Virtual cluster applications often involve the problem of large file transfer between physical clusters. In addition, with the wide spread of computer and Internet technology, large amounts of data are generated every day. Grid computing technology also makes large-scale data-intensive applications widely developed. In such applications, big data's generation cluster. The storage cluster and the processing cluster are often distributed in the physical location far from each other. These data files are collected from the remote data collection point to the processing center for calculation, display and storage. Virtual cluster and data-intensive parallel processing need to transfer mass data efficiently in WAN distributed shared computing environment. We hope that the big data file can be transferred to the corresponding processing node in as short a time as possible, so that data processing can be carried out concurrently. Therefore, how to quickly transfer large files between multiple clusters. Gradually become the hot spot of research. At present, many domestic and foreign experts and scholars have carried on the related research to the parallel processing big file transfer algorithm, respectively from the scheduling policy and the routing policy two aspects. This paper proposes some application techniques to improve the performance of file transfer. At present, the research of file transfer algorithm focuses on improving the parallelism of transmission and the utilization of network bandwidth. The whole transmission time of batch file request is shortened. Most of the current work uses multi-path transmission, multi-hop path transmission and multi-copy method to realize file parallel transmission. However, the ability of receiving files in parallel is not taken into account, and only in this way can the transmission parallelism be improved. Some current research work has not considered the global control and conflict coordination of batch file request transmission throughout the network. In this paper, an on-demand parallel file transfer algorithm, OFPT1, is proposed to support virtual cluster migration. On-demand File Parallel transfer. The purpose of OFPT algorithm is to minimize the overall completion time of batch file transfer request. The algorithm is based on the characteristics of fast data transfer within the cluster. The destination end is extended to all the nodes with external connection in the cluster to realize the parallel of the destination end and to disperse the transmission load of the single node. In the transmission path, the multi-path is used to realize the parallel transmission, and for the single path, the multi-path is used to realize the parallel transmission. The method of multi-hop path hashing is used and the number of hops of the path is adjusted flexibly to obtain the optimal transmission path. In order to solve the bandwidth conflict between the paths of multiple requests the bandwidth utilization of network resources can be improved and the batch file requests can be quickly realized by allocating the network bandwidth according to the global demand. In this paper, NS2 simulation software is used to simulate the experimental environment of batch large file transfer among clusters, and detailed experiments have been carried out, and the transfer mode and the number of file copies have been tested in turn. Finally, the experimental results show that the on-demand parallel file transfer algorithm OFPT can effectively improve the utilization of network resources. The transmission performance is obviously better than the current large file transfer algorithm in WAN, which is up to the expectation of this paper.
【學(xué)位授予單位】：吉林大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2013
【分類(lèi)號(hào)】：TP393.093;TP338.6

【共引文獻(xiàn)】

相關(guān)碩士學(xué)位論文 前1條

1 劉宏亮;BitTorrent核心算法研究與改進(jìn)[D];北京交通大學(xué);2008年

本文編號(hào)：1432526