大規(guī)模時間序列數據存儲系統(tǒng)的研究與實現
發(fā)布時間：2018-08-14 18:30
【摘要】：時間序列數據，即一段時間內以固定的時間間隔采集的數據點的序列，，已成為生產生活中重要的信息記錄形式。RRDtool是當今被廣泛使用的一種存儲時間序列數據的數據庫工具，但是基于RRDtool構建的時間序列數據存儲系統(tǒng)在處理大規(guī)模數據業(yè)務時I/O負荷較重，單位時間內能夠處理的RRD文件數量不能滿足需求等問題；同時，數據量的迅速增長要求存儲系統(tǒng)的容量具有較好的擴展性，特別是能夠在不影響系統(tǒng)在線工作的情況下調整存儲容量；另一方面，考慮到系統(tǒng)可能發(fā)生異�；虿糠质転模枰环N在這些情況下能保證系統(tǒng)可用性的存儲方案。 針對上述問題，本文研究并實現了一種面向大規(guī)模時間序列數據的存儲系統(tǒng)。構建該系統(tǒng)的關鍵部分是mem-RRD和MooseFS，前者是對RRDtool的改進實現，I/O性能更好；后者是一種分布式文件系統(tǒng)，可以保證存儲系統(tǒng)的可用性和擴展性。 本文首先介紹了一種命名為mem-RRD的基于用戶空間緩沖的RRDtool改進方案的設計和實現過程；然后給出利用mem-RRD和MooseFS構建和部署面向大規(guī)模時間序列數據的存儲系統(tǒng)的方案，最后針對該存儲系統(tǒng)的I/O性能、可用性和擴展性進行了詳細的測試，并對測試數據進行對比和分析。測試結果證明，基于mem-RRD和MooseFS構建的面向大規(guī)模時間序列數據的存儲系統(tǒng)在I/O性能、可用性、容量擴展性等方面都有較大改進或較好表現。
[Abstract]:Time series data, which is the sequence of data points collected at a fixed time interval in a certain period of time, has become an important record form of information in production and life. RRDtool is a widely used database tool for storing time series data. However, the time series data storage system based on RRDtool is heavily loaded with I / O, and the number of RRD files per unit time can not meet the requirements. The rapid growth of data requires better scalability of storage systems, in particular the ability to adjust storage capacity without affecting the online operation of the system, and, on the other hand, considering that the system may be anomalous or partially affected, There is a need for a storage scheme that ensures the availability of the system in these cases. In order to solve the above problems, a large-scale time series data storage system is studied and implemented in this paper. The key part of this system is mem-RRD, which is a better implementation of I / O for RRDtool, and MooseFS, which is a distributed file system that can guarantee the availability and scalability of storage system. This paper first introduces the design and implementation of an improved RRDtool scheme based on user space buffer named mem-RRD, and then gives a scheme to build and deploy a storage system for large-scale time series data using mem-RRD and MooseFS. Finally, the I / O performance, availability and scalability of the storage system are tested in detail, and the test data are compared and analyzed. The test results show that the storage system for large-scale time series data based on mem-RRD and MooseFS has great improvement or good performance in the aspects of I / O performance, availability, capacity expansion and so on.
【學位授予單位】：華中科技大學
【學位級別】：碩士
【學位授予年份】：2013
【分類號】：TP333
本文編號：2183738
[Abstract]:Time series data, which is the sequence of data points collected at a fixed time interval in a certain period of time, has become an important record form of information in production and life. RRDtool is a widely used database tool for storing time series data. However, the time series data storage system based on RRDtool is heavily loaded with I / O, and the number of RRD files per unit time can not meet the requirements. The rapid growth of data requires better scalability of storage systems, in particular the ability to adjust storage capacity without affecting the online operation of the system, and, on the other hand, considering that the system may be anomalous or partially affected, There is a need for a storage scheme that ensures the availability of the system in these cases. In order to solve the above problems, a large-scale time series data storage system is studied and implemented in this paper. The key part of this system is mem-RRD, which is a better implementation of I / O for RRDtool, and MooseFS, which is a distributed file system that can guarantee the availability and scalability of storage system. This paper first introduces the design and implementation of an improved RRDtool scheme based on user space buffer named mem-RRD, and then gives a scheme to build and deploy a storage system for large-scale time series data using mem-RRD and MooseFS. Finally, the I / O performance, availability and scalability of the storage system are tested in detail, and the test data are compared and analyzed. The test results show that the storage system for large-scale time series data based on mem-RRD and MooseFS has great improvement or good performance in the aspects of I / O performance, availability, capacity expansion and so on.
【學位授予單位】：華中科技大學
【學位級別】：碩士
【學位授予年份】：2013
【分類號】：TP333
【參考文獻】
相關期刊論文 前10條
1 趙澄東;王艷麗;趙保成;;信息技術及信息技術產業(yè)化分析[J];福建電腦;2009年06期
2 楊岳湘;鄧文平;鄧勁生;李陽;;基于云存儲的網盤系統(tǒng)架構及關鍵技術研究[J];電信科學;2012年10期
3 沈俊 ,顧冠群 ,羅軍舟;網絡管理的研究和發(fā)展[J];計算機研究與發(fā)展;2002年10期
4 楊德志,黃華,張建剛,許魯;大容量、高性能、高擴展能力的藍鯨分布式文件系統(tǒng)[J];計算機研究與發(fā)展;2005年06期
5 張裔智;趙毅;湯小斌;;MD5算法研究[J];計算機科學;2008年07期
6 王珊;肖艷芹;劉大為;覃雄派;;內存數據庫關鍵技術研究[J];計算機應用;2007年10期
7 周豐;逆波蘭表達式及其算法實現[J];武漢交通職業(yè)學院學報;2004年02期
8 唐海娜,李俊;基于RRD的網絡流量監(jiān)測方法[J];微電子學與計算機;2003年07期
9 吳綱;;RRDtool性能優(yōu)化的研究與實現[J];襄樊職業(yè)技術學院學報;2008年04期
10 畢建平,謝萍,劉艷萍;網絡文件系統(tǒng)NFS[J];自動化技術與應用;2001年02期
相關博士學位論文 前1條
1 錢迎進;大規(guī)模Lustre集群文件系統(tǒng)關鍵技術的研究[D];國防科學技術大學;2011年
本文編號：2183738
本文鏈接：http://sikaile.net/kejilunwen/jisuanjikexuelunwen/2183738.html
