分布式海量數(shù)據(jù)儲存系統(tǒng)負載均衡算法的優(yōu)化設(shè)計與實現(xiàn)
發(fā)布時間：2018-10-31 09:27
【摘要】：信息社會給人們帶來便利的同時也產(chǎn)生了大量的數(shù)據(jù)，關(guān)系型數(shù)據(jù)庫由于其架構(gòu)缺陷不適用于處理大數(shù)據(jù)。目前比較常用的存儲和處理海量數(shù)據(jù)的系統(tǒng)是是NoSQL(Not only Structured Query Language)數(shù)據(jù)庫。顯然，如何協(xié)同這些獨立的計算機高效地完成任務(wù)是十分重要的，這就是負載均衡的作用。負載均衡是既定硬件架構(gòu)下整個系統(tǒng)性能的關(guān)鍵，近些年來隨著云計算的興起迅速成為研究熱點。 論文在研究了常用的NoSQL系統(tǒng)和負載均衡技術(shù)的基礎(chǔ)上，設(shè)計了一種新的負載均衡算法。這種算法綜合利用了大數(shù)據(jù)處理系統(tǒng)本身的特點，具有較好的性能。同時該算法還能在最優(yōu)節(jié)點已經(jīng)被占用的情況下迅速找出次最優(yōu)節(jié)點等，非常適合用于高并發(fā)系統(tǒng)的負載均衡。由于內(nèi)存比磁盤的尋道速度大約快十萬倍，為了提高響應(yīng)速度，人們總是希望將盡可能多的數(shù)據(jù)由磁盤提前加載進入內(nèi)存。但內(nèi)存容量比磁盤小很多，因此內(nèi)存容量往往成為大數(shù)據(jù)處理系統(tǒng)的瓶頸。為了提高內(nèi)存的使用效率，本文提出了一種類似內(nèi)存池的方法來重寫管理內(nèi)存分配的配置器，使內(nèi)存的有效使用率得到了很大的改善。本文的主要工作包括： 首先，研究了常用的NoSQL數(shù)據(jù)庫和負載均衡算法，設(shè)計和實現(xiàn)了一個效率較高的負載均衡算法。通過引入節(jié)點與兄弟節(jié)點的子節(jié)點的關(guān)系，并輔之以旋轉(zhuǎn)操作，有效降低了整棵排序二叉樹的高度，使算法的時間代價從常用的加權(quán)輪詢算法的O (N)降為O (logN)，而空間代價保持在O (N)不變。論文中除了提供算法實現(xiàn)的偽代碼以外，，還對算法進行了分析并提供了流程圖，可以根據(jù)流程圖方便地移植到其他相似的系統(tǒng)中。 配置器負責內(nèi)存的分配和回收，默認的配置器具有效率不高、易產(chǎn)生內(nèi)存碎片等弱點。本文通過采用內(nèi)存池的思想優(yōu)化了配置器，通過一次性從內(nèi)核申請較大的內(nèi)存并自己維護，提高了內(nèi)存分配和使用的效率。此外，還根據(jù)不同的場景實現(xiàn)了多種配置器，并實現(xiàn)了配置器的參數(shù)化設(shè)置，使配置器可以根據(jù)使用場景通過腳本實現(xiàn)不同的配置，從而可以動態(tài)地對系統(tǒng)進行調(diào)節(jié)。 最后，搭建了測試環(huán)境，并對該算法和內(nèi)存優(yōu)化進行了測試并對全文進行了總結(jié)和展望。
[Abstract]:The information society brings convenience to people but also produces a lot of data. Relational database is not suitable for dealing with big data because of its structure defect. At present, the commonly used system for storing and processing massive data is NoSQL (Not only Structured Query Language) database. Obviously, it is very important to cooperate with these independent computers to accomplish the task efficiently, which is the function of load balancing. Load balancing is the key to the performance of the whole system under the established hardware architecture. In recent years, with the rise of cloud computing, it has become a research hotspot. Based on the research of NoSQL system and load balancing technology, a new load balancing algorithm is designed in this paper. This algorithm makes full use of the characteristics of big data processing system and has good performance. At the same time, the algorithm can quickly find the sub-optimal nodes when the optimal nodes have been occupied, so it is very suitable for load balancing in high concurrent systems. Because the memory is about 100,000 times faster than the disk, in order to improve the response speed, people always want to load as much data as possible from the disk into memory. But memory capacity is much smaller than disk, so memory capacity often becomes the bottleneck of big data processing system. In order to improve the efficiency of memory use, this paper proposes a method similar to memory pool to rewrite the configuration that manages memory allocation, which greatly improves the effective utilization of memory. The main work of this paper is as follows: firstly, the common NoSQL database and load balancing algorithm are studied, and an efficient load balancing algorithm is designed and implemented. By introducing the relationship between the nodes and the children of the brother nodes, and adding the rotation operation, the height of the whole sorting binary tree is reduced effectively, and the time cost of the algorithm is reduced from the O (N) of the commonly used weighted polling algorithm to the O (logN),. The cost of space remains unchanged at O (N). In addition to the pseudo code of the algorithm, the algorithm is analyzed and the flow chart is provided, which can be easily transplanted to other similar systems according to the flow chart. The configurator is responsible for memory allocation and recovery. The default configuration is inefficient and prone to memory fragmentation and other weaknesses. This paper optimizes the configuration by using the idea of memory pool, and improves the efficiency of memory allocation and usage by applying large memory from the kernel at one time and maintaining itself. In addition, a variety of configurators are implemented according to different scenarios, and the parameterized settings of the configurators are implemented, so that the configurators can realize different configurations through scripts according to the usage scenarios, so that the system can be dynamically adjusted. Finally, the test environment is built, and the algorithm and memory optimization are tested, and the full text is summarized and prospected.
【學位授予單位】：電子科技大學
【學位級別】：碩士
【學位授予年份】：2013
【分類號】：TP333;TP311.13
本文編號：2301666
[Abstract]:The information society brings convenience to people but also produces a lot of data. Relational database is not suitable for dealing with big data because of its structure defect. At present, the commonly used system for storing and processing massive data is NoSQL (Not only Structured Query Language) database. Obviously, it is very important to cooperate with these independent computers to accomplish the task efficiently, which is the function of load balancing. Load balancing is the key to the performance of the whole system under the established hardware architecture. In recent years, with the rise of cloud computing, it has become a research hotspot. Based on the research of NoSQL system and load balancing technology, a new load balancing algorithm is designed in this paper. This algorithm makes full use of the characteristics of big data processing system and has good performance. At the same time, the algorithm can quickly find the sub-optimal nodes when the optimal nodes have been occupied, so it is very suitable for load balancing in high concurrent systems. Because the memory is about 100,000 times faster than the disk, in order to improve the response speed, people always want to load as much data as possible from the disk into memory. But memory capacity is much smaller than disk, so memory capacity often becomes the bottleneck of big data processing system. In order to improve the efficiency of memory use, this paper proposes a method similar to memory pool to rewrite the configuration that manages memory allocation, which greatly improves the effective utilization of memory. The main work of this paper is as follows: firstly, the common NoSQL database and load balancing algorithm are studied, and an efficient load balancing algorithm is designed and implemented. By introducing the relationship between the nodes and the children of the brother nodes, and adding the rotation operation, the height of the whole sorting binary tree is reduced effectively, and the time cost of the algorithm is reduced from the O (N) of the commonly used weighted polling algorithm to the O (logN),. The cost of space remains unchanged at O (N). In addition to the pseudo code of the algorithm, the algorithm is analyzed and the flow chart is provided, which can be easily transplanted to other similar systems according to the flow chart. The configurator is responsible for memory allocation and recovery. The default configuration is inefficient and prone to memory fragmentation and other weaknesses. This paper optimizes the configuration by using the idea of memory pool, and improves the efficiency of memory allocation and usage by applying large memory from the kernel at one time and maintaining itself. In addition, a variety of configurators are implemented according to different scenarios, and the parameterized settings of the configurators are implemented, so that the configurators can realize different configurations through scripts according to the usage scenarios, so that the system can be dynamically adjusted. Finally, the test environment is built, and the algorithm and memory optimization are tested, and the full text is summarized and prospected.
【學位授予單位】：電子科技大學
【學位級別】：碩士
【學位授予年份】：2013
【分類號】：TP333;TP311.13
【參考文獻】
相關(guān)期刊論文 前1條
1 楊際祥;譚國真;王榮生;;并行與分布式計算動態(tài)負載均衡策略綜述[J];電子學報;2010年05期
本文編號：2301666
本文鏈接：http://sikaile.net/kejilunwen/jisuanjikexuelunwen/2301666.html
最近更新
教材專著
