本文選題：分布式 + KEYVALUE�。� 參考：《復(fù)旦大學》2013年碩士論文

【摘要】：現(xiàn)行的互聯(lián)網(wǎng)企業(yè)數(shù)據(jù)中心擁有龐大的計算機集群,而每天產(chǎn)生的日志信息為海量(TB)數(shù)據(jù),對于管理人員來說,如此海量的數(shù)據(jù)的的持久化以及搜索將是一個巨大的問題,基于ACID理論的關(guān)系數(shù)據(jù)庫無法滿足現(xiàn)行互聯(lián)網(wǎng)數(shù)據(jù)管理的需求。而隨著NOSQL技術(shù)的興起,NOSQL數(shù)據(jù)能夠很好的實現(xiàn)數(shù)據(jù)存儲的橫向和縱向擴展。 基于BASE理論的KEYVALUE數(shù)據(jù)庫選用AP為理論基礎(chǔ),在保證最終一致性的基礎(chǔ)上,能夠很好的滿足可用性和分區(qū)容忍性。因為對很多特定的系統(tǒng)來說,絕大部分的檢索都是基于主鍵的查詢,因此KEYVALUE數(shù)據(jù)庫存儲將會是一個很好的選擇,而另一方面,大量互聯(lián)網(wǎng)數(shù)據(jù)在管理上并不需要保證強一致性,而只需要保證最終一致性。同時,KEYVALUE型數(shù)據(jù)庫具有極高并發(fā)讀寫性能,能夠很好的橫向及縱向擴展、能夠滿足當前互聯(lián)網(wǎng)應(yīng)用中的海量數(shù)據(jù)的存儲、管理、搜索、統(tǒng)計,具有高可靠性及高擴展性。 本論文在研究現(xiàn)有流行的KEYVALUE數(shù)據(jù)庫的設(shè)計原理、應(yīng)用環(huán)境、特點及應(yīng)用案例的基礎(chǔ)上,結(jié)合海量日志型數(shù)據(jù)的特點,測試現(xiàn)有KEYVALUE數(shù)據(jù)庫插入、搜索及持久化性能,在memlink基礎(chǔ)上設(shè)計出數(shù)據(jù)的多級持久化方案,同時在分布式策略上使用ZEROMQ為基礎(chǔ),提出海量日志分布式存儲系統(tǒng)架構(gòu)方案、設(shè)計方案、部署策略,整個系統(tǒng)采用C語言編寫實現(xiàn),并最終完成性能測試工作。從最終的測試結(jié)果來看,本方案能夠很好的降低部署和運行成本,在保證可用性以及最終一致性的前提下,具有極高的性能,即在兼顧效率條件下,本系統(tǒng)完全能夠滿足現(xiàn)有互聯(lián)網(wǎng)海量日志的存儲、查詢、統(tǒng)計等各項需求。
[Abstract]:The current Internet enterprise data center has a huge computer cluster, and the daily log information generated is a huge amount of TB-based data. For managers, the persistence and search of such huge amounts of data will be a huge problem. The relational database based on ACID theory can not meet the demand of current Internet data management. With the rise of NOSQL technology, NOSQL data can achieve the horizontal and vertical expansion of data storage. The KEYVALUE database based on BASE theory is based on AP, which can satisfy the availability and partition tolerance well on the basis of ensuring the final consistency. For many specific systems, most searches are based on primary key queries, so KEYVALUE database storage will be a good choice. On the other hand, there is no need to ensure strong consistency in the management of a large amount of Internet data. You just need to ensure final consistency. At the same time, KEYVALUE database has the extremely high concurrent reading and writing performance, can very good horizontal and vertical expansion, can satisfy the massive data storage, the management, the search, the statistics in the current Internet application, has the high reliability and the high expansibility. On the basis of studying the design principle, application environment, characteristics and application cases of the popular KEYVALUE database, this paper tests the insert, search and persistence performance of the existing KEYVALUE database based on the characteristics of massive log data. The multilevel persistence scheme of data is designed on the basis of memlink. At the same time, on the basis of ZEROMQ, the architecture scheme, design scheme and deployment strategy of distributed storage system for mass log are put forward. The whole system is implemented in C language. And the final performance test work completed. From the final test results, this scheme can reduce the cost of deployment and operation very well, and it has a very high performance under the premise of ensuring availability and final consistency, that is, under the condition of taking into account the efficiency. The system can fully meet the existing Internet massive log storage, query, statistics and other needs.
【學位授予單位】：復(fù)旦大學
【學位級別】：碩士
【學位授予年份】：2013
【分類號】：TP333

【參考文獻】

相關(guān)期刊論文 前1條

1 馮登國;張敏;張妍;徐震;;云計算安全研究[J];軟件學報;2011年01期

，

本文編號：1825569