發(fā)布時間：2018-04-04 20:20

  本文選題：分布式文件系統(tǒng)　切入點：海量小文件　出處：《南京大學》2017年碩士論文

【摘要】：近年來,電信運營商傳統(tǒng)業(yè)務(wù)受到了互聯(lián)網(wǎng)浪潮的猛烈沖擊,運營商不得不尋求轉(zhuǎn)型,以適應(yīng)互聯(lián)網(wǎng)的發(fā)展潮流。運營商在發(fā)展動漫基地和音樂基地業(yè)務(wù)時,隨著業(yè)務(wù)的發(fā)展,產(chǎn)生的數(shù)據(jù)日益增多,產(chǎn)生了海量的小文件,這使得傳統(tǒng)文件系統(tǒng)面臨巨大的挑戰(zhàn),因此解決海量小文件存儲的問題具有非常重要的意義。目前存在的分布式文件系統(tǒng)可以實現(xiàn)海量文件的存儲,但是同時也存在一些痛點。首先,現(xiàn)存的分布式文件系統(tǒng)主要提供的核心功能是文件的上傳、下載、更新和刪除。其次,目前的分布式文件系統(tǒng)存在的不足主要是無法對海量小文件的元數(shù)據(jù)進行有效的管理以及系統(tǒng)架構(gòu)過于復(fù)雜。電信運營商需要一個能夠擁有核心功能,又能克服這些不足的文件存儲系統(tǒng)。本論文設(shè)計實現(xiàn)了統(tǒng)一的文件存儲系統(tǒng),該系統(tǒng)主要為電信運營商的動漫基地和音樂基地業(yè)務(wù)提供數(shù)據(jù)存儲服務(wù)。整個系統(tǒng)主要由MetadataServer、BlockServer及通信模塊組成。系統(tǒng)通過使用Netty框架為業(yè)務(wù)提供高并發(fā)、高可用的存儲訪問服務(wù);通過文件多副本存儲,系統(tǒng)在提供良好讀寫性能的基礎(chǔ)上,也提供了高可用性的保障;通過將元數(shù)據(jù)管理和存儲數(shù)據(jù)訪問分離,來減小元數(shù)據(jù)管理機器的壓力,提升元數(shù)據(jù)管理效率;為了減少inode資源的占用,系統(tǒng)采用小文件合并的方式,大大減少了存儲文件所需要的inode資源,使得元數(shù)據(jù)信息可以全部緩存到內(nèi)存中,提升存儲效率和訪問效率。通信模塊為整個系統(tǒng)提供高性能的通信服務(wù),通信模塊包括內(nèi)部通信和外部通信。內(nèi)部通信為節(jié)點之間提供穩(wěn)定的通信服務(wù),外部通信使得用戶通過HTTP和HTTPS協(xié)議訪問系統(tǒng)服務(wù)。通過本文設(shè)計的海量小文件系統(tǒng),用戶可以有效的對海量小文件進行存儲及管理。目前整個系統(tǒng)已經(jīng)上線,為電信運營商提供了高效、可靠的數(shù)據(jù)存儲服務(wù)。
[Abstract]:In recent years, the traditional business of telecom operators has been severely impacted by the tide of the Internet, operators have to seek transformation to adapt to the trend of development of the Internet.Operators in the development of animation base and music base business, with the development of business, the data generated by the increasing, produced a large number of small files, which makes the traditional file system is facing a huge challenge.Therefore, it is of great significance to solve the problem of storing large amounts of small files.The existing distributed file systems can store large amount of files, but there are also some pain points.First of all, the core functions of the existing distributed file system are file upload, download, update and delete.Secondly, the shortcomings of current distributed file systems are that they can not effectively manage the metadata of a large number of small files and the system architecture is too complex.Telecom operators need a file storage system that has core functions and can overcome these shortcomings.This paper designs and implements a unified file storage system, which mainly provides data storage services for the business of cartoon and music base of telecom operators.The whole system is mainly composed of Metadata Server Block Server and communication module.The system provides high concurrency and high availability storage access services for business by using Netty framework, and provides high availability on the basis of providing good read and write performance through file multiple replica storage.In order to reduce the pressure of metadata management machine and improve the efficiency of metadata management, the system adopts the method of small file merging in order to reduce the occupation of inode resources.The inode resources needed to store files are greatly reduced, so that metadata information can be completely cached into memory, thus improving storage efficiency and access efficiency.The communication module provides high performance communication service for the whole system. The communication module includes internal communication and external communication.Internal communication provides stable communication services between nodes. External communication enables users to access system services via HTTP and HTTPS protocols.Through the massive file system designed in this paper, users can effectively store and manage the massive small files.At present, the whole system has been online, providing efficient and reliable data storage service for telecom operators.
【學位授予單位】：南京大學
【學位級別】：碩士
【學位授予年份】：2017
【分類號】：TP333
，

本文編號：1711533