本文選題：并行網(wǎng)絡(luò)文件系統(tǒng) + 負(fù)載均衡　； 參考：《華中科技大學(xué)》2012年碩士論文

【摘要】：負(fù)載均衡技術(shù)是并行文件系統(tǒng)中常見且不可或缺的重要優(yōu)化手段。在并行文件系統(tǒng)中實現(xiàn)副本技術(shù)，利用負(fù)載分析方法和調(diào)度算法，可以有效地將系統(tǒng)中各種負(fù)載均衡地分配到各存儲節(jié)點上，以提高文件系統(tǒng)的可用性、穩(wěn)定性和服務(wù)質(zhì)量。 在并行文件系統(tǒng)中，隨著節(jié)點的增加、刪除，文件的創(chuàng)建、刪除和修改，以及訪問次數(shù)的爆炸性增長，各個存儲節(jié)點的負(fù)載以不可預(yù)知的方式動態(tài)變化，常常發(fā)生部分服務(wù)器資源損耗過度而另外一些服務(wù)器使用率低下的情況，即負(fù)載傾斜。同時，節(jié)點的突發(fā)故障極易導(dǎo)致系統(tǒng)無法正常運行。雖然使用副本機(jī)制有助于解決上述問題，但在不同場景下，還需要針對場景中的I/O行為特征，，以副本為基礎(chǔ)，圍繞負(fù)載均衡技術(shù)進(jìn)行研究，找出適用的策略和時機(jī)。本文針對一種典型的并行文件系統(tǒng)pNFS（Parallel Network File System），設(shè)計并實現(xiàn)了基于概率分布的動態(tài)負(fù)載均衡機(jī)制PDDB（Probability Distribution Dynamic Balance）。 在并行網(wǎng)絡(luò)文件系統(tǒng)中，PDDB通過對文件創(chuàng)建副本，按照容量均衡的方式進(jìn)行文件副本的合理放置，同時設(shè)計了鏡像和交錯的兩種副本放置模式。PDDB在各數(shù)據(jù)服務(wù)器上構(gòu)建了自適應(yīng)的負(fù)載監(jiān)控系統(tǒng)，由元數(shù)據(jù)服務(wù)器收集各個節(jié)點的CPU、內(nèi)存、存儲空間、網(wǎng)絡(luò)帶寬、磁盤帶寬等負(fù)載信息，經(jīng)過對負(fù)載信息的整合處理，并結(jié)合已經(jīng)獲得的歷史數(shù)據(jù)信息，以綜合負(fù)載的大小決定任務(wù)分配的概率，將訪問均衡地分配到當(dāng)前負(fù)載較低的一組服務(wù)器上，并通過熱點遷移調(diào)度各個節(jié)點之間的負(fù)載狀態(tài)，避免群聚效應(yīng)。并加入副本元數(shù)據(jù)的管理以及副本一致性的維護(hù)，保證當(dāng)任一副本的存儲節(jié)點發(fā)生故障時，可以通過其他副本正常獲取文件內(nèi)容，確保系統(tǒng)的正常運行。 在測試中發(fā)現(xiàn)，與使用隨機(jī)算法和最小負(fù)載優(yōu)先算法的負(fù)載均衡機(jī)制相比較，PDDB機(jī)制使得各存儲節(jié)點之間的最大負(fù)載差距比二者分別減少了42%和30%，且在文件系統(tǒng)總負(fù)載相同時，各個節(jié)點的負(fù)載變化也相對平緩，平均負(fù)載至少降低10%以上。文件系統(tǒng)的平均網(wǎng)絡(luò)吞吐率提升20%，可靠性、穩(wěn)定性和可擴(kuò)展性均得到改善。
[Abstract]:Load balancing is a common and indispensable optimization method in parallel file systems. By using load analysis method and scheduling algorithm, replicas in parallel file systems can be effectively distributed to each storage node in order to improve the availability, stability and quality of service of the file system. In parallel file system, with the increase of nodes, deletion, creation, deletion and modification of files, and the explosive increase of access, the load of each storage node changes in an unpredictable way. Some server resources are overused and others are underutilized, that is, load tilting. At the same time, the sudden failure of nodes can easily lead to the normal operation of the system. Although using replica mechanism is helpful to solve the above problems, it is necessary to study the load balancing technology based on replicas in different scenarios, and find out the appropriate strategies and opportunities according to the behavior characteristics of I / O in different scenarios. For a typical parallel file system, pNFS(Parallel Network File system, a dynamic load balancing mechanism based on probability distribution, PDDB(Probability Distribution Dynamic balance, is designed and implemented in this paper. In parallel network file system, PDDB can make a copy of the file by making a copy of the file, and make a reasonable arrangement of the copy of the file according to the way of capacity balance. At the same time, two replica placement modes, mirroring and interleaving, are designed. PDDB constructs an adaptive load monitoring system on each data server. The metadata server collects the CPU, memory, storage space and network bandwidth of each node. The load information, such as disk bandwidth, is processed by the integration of the load information, and combined with the historical data obtained, the probability of task allocation is determined by the size of the comprehensive load. Access is distributed evenly to a group of servers with low current load, and the load state between nodes is scheduled by hot spot migration to avoid clustering effect. The management of replica metadata and the maintenance of replica consistency are added to ensure that the contents of files can be obtained through other replicas when any storage node of the replica fails to ensure the normal operation of the system. It was found in the test that the PDDB mechanism reduced the maximum load gap between storage nodes by 42% and 30% respectively compared with the load balancing mechanism using random algorithm and minimum load first algorithm, and when the total load of the file system was the same, the PDDB mechanism reduced the maximum load gap between each storage node by 42% and 30% respectively, and when the total load of the file system was the same, The load change of each node is relatively smooth, the average load is reduced by more than 10%. The average network throughput of the file system is increased by 20%, and the reliability, stability and scalability are improved.
【學(xué)位授予單位】：華中科技大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2012
【分類號】：TP338.6

【參考文獻(xiàn)】

相關(guān)期刊論文 前4條

1 龐麗萍,許俊,徐婕,岳建輝;PVFS數(shù)據(jù)訪問的負(fù)載平衡[J];華中科技大學(xué)學(xué)報(自然科學(xué)版);2004年07期

2 龔梅;王鵬;吳躍;;一種集群系統(tǒng)的透明動態(tài)反饋負(fù)載均衡算法[J];計算機(jī)應(yīng)用;2007年11期

3 張媛;于冠龍;盧澤新;劉亞萍;;并行網(wǎng)絡(luò)文件系統(tǒng)PNFS性能評測與分析[J];計算機(jī)工程與應(yīng)用;2009年35期

4 陳志剛,李登,曾志文;分布式系統(tǒng)中一種動態(tài)負(fù)載均衡策略、相關(guān)模型及算法研究[J];小型微型計算機(jī)系統(tǒng);2002年12期

本文編號：1862609