【摘要】：信息社會(huì)給人們帶來(lái)便利的同時(shí)也產(chǎn)生了大量的數(shù)據(jù)，關(guān)系型數(shù)據(jù)庫(kù)由于其架構(gòu)缺陷不適用于處理大數(shù)據(jù)。目前比較常用的存儲(chǔ)和處理海量數(shù)據(jù)的系統(tǒng)是是NoSQL(Not only Structured Query Language)數(shù)據(jù)庫(kù)。顯然，如何協(xié)同這些獨(dú)立的計(jì)算機(jī)高效地完成任務(wù)是十分重要的，這就是負(fù)載均衡的作用。負(fù)載均衡是既定硬件架構(gòu)下整個(gè)系統(tǒng)性能的關(guān)鍵，近些年來(lái)隨著云計(jì)算的興起迅速成為研究熱點(diǎn)。 論文在研究了常用的NoSQL系統(tǒng)和負(fù)載均衡技術(shù)的基礎(chǔ)上，設(shè)計(jì)了一種新的負(fù)載均衡算法。這種算法綜合利用了大數(shù)據(jù)處理系統(tǒng)本身的特點(diǎn)，具有較好的性能。同時(shí)該算法還能在最優(yōu)節(jié)點(diǎn)已經(jīng)被占用的情況下迅速找出次最優(yōu)節(jié)點(diǎn)等，非常適合用于高并發(fā)系統(tǒng)的負(fù)載均衡。由于內(nèi)存比磁盤的尋道速度大約快十萬(wàn)倍，為了提高響應(yīng)速度，人們總是希望將盡可能多的數(shù)據(jù)由磁盤提前加載進(jìn)入內(nèi)存。但內(nèi)存容量比磁盤小很多，因此內(nèi)存容量往往成為大數(shù)據(jù)處理系統(tǒng)的瓶頸。為了提高內(nèi)存的使用效率，本文提出了一種類似內(nèi)存池的方法來(lái)重寫管理內(nèi)存分配的配置器，使內(nèi)存的有效使用率得到了很大的改善。本文的主要工作包括： 首先，研究了常用的NoSQL數(shù)據(jù)庫(kù)和負(fù)載均衡算法，設(shè)計(jì)和實(shí)現(xiàn)了一個(gè)效率較高的負(fù)載均衡算法。通過(guò)引入節(jié)點(diǎn)與兄弟節(jié)點(diǎn)的子節(jié)點(diǎn)的關(guān)系，并輔之以旋轉(zhuǎn)操作，有效降低了整棵排序二叉樹的高度，使算法的時(shí)間代價(jià)從常用的加權(quán)輪詢算法的O (N)降為O (logN)，而空間代價(jià)保持在O (N)不變。論文中除了提供算法實(shí)現(xiàn)的偽代碼以外，，還對(duì)算法進(jìn)行了分析并提供了流程圖，可以根據(jù)流程圖方便地移植到其他相似的系統(tǒng)中。 配置器負(fù)責(zé)內(nèi)存的分配和回收，默認(rèn)的配置器具有效率不高、易產(chǎn)生內(nèi)存碎片等弱點(diǎn)。本文通過(guò)采用內(nèi)存池的思想優(yōu)化了配置器，通過(guò)一次性從內(nèi)核申請(qǐng)較大的內(nèi)存并自己維護(hù)，提高了內(nèi)存分配和使用的效率。此外，還根據(jù)不同的場(chǎng)景實(shí)現(xiàn)了多種配置器，并實(shí)現(xiàn)了配置器的參數(shù)化設(shè)置，使配置器可以根據(jù)使用場(chǎng)景通過(guò)腳本實(shí)現(xiàn)不同的配置，從而可以動(dòng)態(tài)地對(duì)系統(tǒng)進(jìn)行調(diào)節(jié)。 最后，搭建了測(cè)試環(huán)境，并對(duì)該算法和內(nèi)存優(yōu)化進(jìn)行了測(cè)試并對(duì)全文進(jìn)行了總結(jié)和展望。
[Abstract]:The information society brings convenience to people but also produces a lot of data. Relational database is not suitable for dealing with big data because of its structure defect. At present, the commonly used system for storing and processing massive data is NoSQL (Not only Structured Query Language) database. Obviously, it is very important to cooperate with these independent computers to accomplish the task efficiently, which is the function of load balancing. Load balancing is the key to the performance of the whole system under the established hardware architecture. In recent years, with the rise of cloud computing, it has become a research hotspot. Based on the research of NoSQL system and load balancing technology, a new load balancing algorithm is designed in this paper. This algorithm makes full use of the characteristics of big data processing system and has good performance. At the same time, the algorithm can quickly find the sub-optimal nodes when the optimal nodes have been occupied, so it is very suitable for load balancing in high concurrent systems. Because the memory is about 100,000 times faster than the disk, in order to improve the response speed, people always want to load as much data as possible from the disk into memory. But memory capacity is much smaller than disk, so memory capacity often becomes the bottleneck of big data processing system. In order to improve the efficiency of memory use, this paper proposes a method similar to memory pool to rewrite the configuration that manages memory allocation, which greatly improves the effective utilization of memory. The main work of this paper is as follows: firstly, the common NoSQL database and load balancing algorithm are studied, and an efficient load balancing algorithm is designed and implemented. By introducing the relationship between the nodes and the children of the brother nodes, and adding the rotation operation, the height of the whole sorting binary tree is reduced effectively, and the time cost of the algorithm is reduced from the O (N) of the commonly used weighted polling algorithm to the O (logN),. The cost of space remains unchanged at O (N). In addition to the pseudo code of the algorithm, the algorithm is analyzed and the flow chart is provided, which can be easily transplanted to other similar systems according to the flow chart. The configurator is responsible for memory allocation and recovery. The default configuration is inefficient and prone to memory fragmentation and other weaknesses. This paper optimizes the configuration by using the idea of memory pool, and improves the efficiency of memory allocation and usage by applying large memory from the kernel at one time and maintaining itself. In addition, a variety of configurators are implemented according to different scenarios, and the parameterized settings of the configurators are implemented, so that the configurators can realize different configurations through scripts according to the usage scenarios, so that the system can be dynamically adjusted. Finally, the test environment is built, and the algorithm and memory optimization are tested, and the full text is summarized and prospected.
【學(xué)位授予單位】：電子科技大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2013
【分類號(hào)】：TP333;TP311.13

【參考文獻(xiàn)】

相關(guān)期刊論文 前1條

1 楊際祥;譚國(guó)真;王榮生;;并行與分布式計(jì)算動(dòng)態(tài)負(fù)載均衡策略綜述[J];電子學(xué)報(bào);2010年05期

本文編號(hào)：2301666