面向海量用戶(hù)的云存儲(chǔ)系統(tǒng)的設(shè)計(jì)與優(yōu)化
發(fā)布時(shí)間：2018-11-08 18:05
【摘要】：隨著信息技術(shù)不斷進(jìn)步、移動(dòng)網(wǎng)絡(luò)的普及,個(gè)人用戶(hù)的數(shù)據(jù)量迅猛增長(zhǎng),對(duì)云存儲(chǔ)服務(wù)需求度急增,眾多知名企業(yè)紛紛投入到個(gè)人用戶(hù)云存儲(chǔ)服務(wù)的研發(fā)與運(yùn)營(yíng)中,如Google, Microsoft, Drupbox,聯(lián)想,金山,華為,百度,電信運(yùn)營(yíng)商等。在這些優(yōu)秀產(chǎn)品中,大多數(shù)采用的策略均是以Hadoop HDFS作為基礎(chǔ)文件系統(tǒng)進(jìn)行二次定制開(kāi)發(fā)。開(kāi)源的Hadoop HDFS以其優(yōu)異的架構(gòu)設(shè)計(jì)與高可擴(kuò)展性、可用性、可靠性、容錯(cuò)性、經(jīng)濟(jì)性及出色的性能風(fēng)靡全球,成為了熱點(diǎn)研究領(lǐng)域。然而,HDFS自身尚有諸多不足尚待解決,如NameNode單點(diǎn)瓶頸、小文件處理能力不足、冗余文件缺乏引用、缺乏用戶(hù)層負(fù)載平衡、不支持文件的斷點(diǎn)續(xù)傳、系統(tǒng)安全性弱、缺乏數(shù)據(jù)加密存儲(chǔ)及共享授權(quán)機(jī)制等問(wèn)題。 HDFS現(xiàn)存的這些不足可以從兩個(gè)方面去彌補(bǔ)：第一,對(duì)HDFS的源碼進(jìn)行修改,即從內(nèi)部對(duì)其進(jìn)行完善；第二,在HDFS上增加一層服務(wù)層,即將部分功能剝離出來(lái),簡(jiǎn)化HDFS本身的功能。第一種方式需要對(duì)HDFS做重大修改,工程量大、難度高、不能向后兼容HDFS版本,并且不能有效解決單點(diǎn)瓶頸、斷點(diǎn)續(xù)傳、文件加密授權(quán)等問(wèn)題。第二種方式束縛條件少,難度低,工程量小,能兼容各種HDFS版本,更重要的是,此方式構(gòu)建的云存儲(chǔ)系統(tǒng)具有很大的改進(jìn)空間,可以解決這些問(wèn)題,具有很強(qiáng)的可擴(kuò)展性,所以本文采用的是后一種方式。 本文著重分析了現(xiàn)有的HDFS架構(gòu),并在此基礎(chǔ)上構(gòu)建了一套面向海量用戶(hù)的云存儲(chǔ)系統(tǒng)架構(gòu),該架構(gòu)為HDFS存在的諸多問(wèn)題給出了優(yōu)化解決方案,并能確保數(shù)據(jù)安全及用戶(hù)隱私保護(hù)。本文的主要?jiǎng)?chuàng)新點(diǎn)如下： 1.提出了一個(gè)基于HDFS的海量用戶(hù)云存儲(chǔ)系統(tǒng)架構(gòu)并分析了此架構(gòu)的優(yōu)勢(shì)：有效地緩解了單點(diǎn)瓶頸問(wèn)題、增強(qiáng)了系統(tǒng)的安全性與可擴(kuò)展性、支持多種訪(fǎng)問(wèn)協(xié)議、兼容HDFS各版本等。 2.提出了一套完整的系統(tǒng)安全保護(hù)機(jī)制：第一,提出了一種能抵抗木馬環(huán)境的客戶(hù)端登錄驗(yàn)證方法以增強(qiáng)用戶(hù)帳戶(hù)的安全性；第二,提出了一種文件的加密存儲(chǔ)與分級(jí)授權(quán)管理辦法以確保用戶(hù)數(shù)據(jù)安全,并能方便文件授權(quán)的分發(fā)與回收；第三,給出了適用于云存儲(chǔ)服務(wù)的訪(fǎng)問(wèn)控制策略,從而能夠更好地保證訪(fǎng)問(wèn)安全。分析表明,該機(jī)制不僅能提升系統(tǒng)的安全性,而且能實(shí)現(xiàn)用戶(hù)隱私保護(hù)及安全訪(fǎng)問(wèn)控制。再配合使用SSL/TLS進(jìn)行加密通信,系統(tǒng)整體安全性得到極大增強(qiáng)。 3.給出了應(yīng)用服務(wù)器間負(fù)載均衡的調(diào)度機(jī)制以實(shí)現(xiàn)用戶(hù)層訪(fǎng)問(wèn)請(qǐng)求的負(fù)載均衡。應(yīng)用服務(wù)器可隨時(shí)加入或退出集群,避免了單點(diǎn)故障問(wèn)題。訪(fǎng)問(wèn)請(qǐng)求的均衡調(diào)度與應(yīng)用服務(wù)器的緩存管理相互配合,能夠有效地提升系統(tǒng)性能及負(fù)載能力。 4.針對(duì)HDFS與海量用戶(hù)特性提出了對(duì)應(yīng)的優(yōu)化方案。例如：增加斷點(diǎn)續(xù)傳功能,對(duì)小文件進(jìn)行打包存儲(chǔ),對(duì)大文件進(jìn)行冗余引用處理,為應(yīng)用服務(wù)器增加緩存,將文件的容器結(jié)構(gòu)與HDFS結(jié)構(gòu)映射等。 與原HDFS系統(tǒng)相比,本文提出的方法在增加系統(tǒng)功能的同時(shí),能夠提升系統(tǒng)的性能及安全性。
[Abstract]:With the development of information technology, the popularity of mobile networks, the rapid increase of the data volume of individual users, the demand for cloud storage services has increased rapidly, and many well-known enterprises have input into the R & D and operation of individual user cloud storage services, such as Google, Microsoft, Drupbox, Lenovo, Kingsoft and Huawei, Baidu, telecom operator, etc. In these excellent products, most of the policies are based on Hadoop HDFS as a base file system for secondary customization development. The open source Hadoop HDFS is a hot research area with its excellent architecture design and high scalability, availability, reliability, fault tolerance, economy, and excellent performance. However, there are many problems still to be solved by the HDFS, such as the single-point bottleneck of the NameNode, the insufficient processing capacity of the small file, the lack of reference of the redundant files, the lack of the load balance of the user, the failure of the file, the weak security of the system, the lack of data encryption and the sharing of the authorization mechanism, etc. The existing shortage of HDFS can be made up of two ways: first, the source code of the HDFS is modified, that is, it is improved from inside; secondly, a layer of service layer is added on the HDFS, the part of the function is peeled off, and the HDFS is simplified Function. In the first way, it is necessary to make major changes to the HDFS. The quantity is large, the difficulty is high, the HDFS version can not be backward compatible, and the single-point bottleneck, the breakpoint continuous transmission, the file encryption authorization and the like cannot be effectively solved. The second way is that the binding condition is small, the difficulty is low, the engineering quantity is small, can be compatible with various HDFS versions, and more importantly, the cloud storage system constructed in this way has a great improvement space, can solve the problems, and has strong expandability, so that the method adopts the following In this paper, the existing HDFS architecture is analyzed, and a set of cloud storage system architecture for mass users is built on this basis. The architecture provides an optimized solution for many problems existing in the HDFS, and can ensure the data security and use. The main part of this paper is to protect the privacy of the household. The innovation point is as follows: 1. A mass user cloud storage system architecture based on HDFS is proposed and the advantages of this architecture are analyzed: the problem of single-point bottleneck is effectively relieved, the security and the expandability of the system are enhanced, a plurality of access protocols are supported, HDFS version and so on. 2. A complete system safety protection mechanism is proposed: first, a client login verification method which can resist the Trojan environment is proposed to enhance the security of the user account; secondly, a file encryption and classification authorization management is proposed The method can ensure the data safety of the user and can facilitate the distribution and recovery of the file authorization; and thirdly, the access control strategy applicable to the cloud storage service is provided, The analysis shows that the system not only can improve the security of the system but also the users Privacy protection and secure access control. Re-use SSL/ TLS for encrypted communications, systems The overall security is greatly enhanced. 3. The scheduling mechanism of the load balance among the application servers is given. the load balancing of the access request of the user layer is realized, and the application server can join or the method comprises the following steps of: quitting the cluster, and avoiding the problem of single point failure. The balancing scheduling of the access request and the cache management of the application server are mutually matched, and the method can improve system performance and load capacity effectively. 4. for HDFS and a corresponding optimization scheme is proposed for the characteristics of the mass user, for example, the breakpoint continuous transmission function is increased, the small file is packaged and stored, the large file is subjected to redundant reference processing, the cache is added to the application server, The container structure of the file and the HDFS structure mapping, etc. Compared with the original HDFS system, the method proposed in this paper is increasing the system
【學(xué)位授予單位】：華東師范大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2013
【分類(lèi)號(hào)】：TP333;TP309
本文編號(hào)：2319262
[Abstract]:With the development of information technology, the popularity of mobile networks, the rapid increase of the data volume of individual users, the demand for cloud storage services has increased rapidly, and many well-known enterprises have input into the R & D and operation of individual user cloud storage services, such as Google, Microsoft, Drupbox, Lenovo, Kingsoft and Huawei, Baidu, telecom operator, etc. In these excellent products, most of the policies are based on Hadoop HDFS as a base file system for secondary customization development. The open source Hadoop HDFS is a hot research area with its excellent architecture design and high scalability, availability, reliability, fault tolerance, economy, and excellent performance. However, there are many problems still to be solved by the HDFS, such as the single-point bottleneck of the NameNode, the insufficient processing capacity of the small file, the lack of reference of the redundant files, the lack of the load balance of the user, the failure of the file, the weak security of the system, the lack of data encryption and the sharing of the authorization mechanism, etc. The existing shortage of HDFS can be made up of two ways: first, the source code of the HDFS is modified, that is, it is improved from inside; secondly, a layer of service layer is added on the HDFS, the part of the function is peeled off, and the HDFS is simplified Function. In the first way, it is necessary to make major changes to the HDFS. The quantity is large, the difficulty is high, the HDFS version can not be backward compatible, and the single-point bottleneck, the breakpoint continuous transmission, the file encryption authorization and the like cannot be effectively solved. The second way is that the binding condition is small, the difficulty is low, the engineering quantity is small, can be compatible with various HDFS versions, and more importantly, the cloud storage system constructed in this way has a great improvement space, can solve the problems, and has strong expandability, so that the method adopts the following In this paper, the existing HDFS architecture is analyzed, and a set of cloud storage system architecture for mass users is built on this basis. The architecture provides an optimized solution for many problems existing in the HDFS, and can ensure the data security and use. The main part of this paper is to protect the privacy of the household. The innovation point is as follows: 1. A mass user cloud storage system architecture based on HDFS is proposed and the advantages of this architecture are analyzed: the problem of single-point bottleneck is effectively relieved, the security and the expandability of the system are enhanced, a plurality of access protocols are supported, HDFS version and so on. 2. A complete system safety protection mechanism is proposed: first, a client login verification method which can resist the Trojan environment is proposed to enhance the security of the user account; secondly, a file encryption and classification authorization management is proposed The method can ensure the data safety of the user and can facilitate the distribution and recovery of the file authorization; and thirdly, the access control strategy applicable to the cloud storage service is provided, The analysis shows that the system not only can improve the security of the system but also the users Privacy protection and secure access control. Re-use SSL/ TLS for encrypted communications, systems The overall security is greatly enhanced. 3. The scheduling mechanism of the load balance among the application servers is given. the load balancing of the access request of the user layer is realized, and the application server can join or the method comprises the following steps of: quitting the cluster, and avoiding the problem of single point failure. The balancing scheduling of the access request and the cache management of the application server are mutually matched, and the method can improve system performance and load capacity effectively. 4. for HDFS and a corresponding optimization scheme is proposed for the characteristics of the mass user, for example, the breakpoint continuous transmission function is increased, the small file is packaged and stored, the large file is subjected to redundant reference processing, the cache is added to the application server, The container structure of the file and the HDFS structure mapping, etc. Compared with the original HDFS system, the method proposed in this paper is increasing the system
【學(xué)位授予單位】：華東師范大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2013
【分類(lèi)號(hào)】：TP333;TP309
【參考文獻(xiàn)】
相關(guān)期刊論文 前10條
1 唐箭;;云存儲(chǔ)系統(tǒng)的分析與應(yīng)用研究[J];電腦知識(shí)與技術(shù);2009年20期
2 許春聰;黃小猛;徐鵬志;吳諾;劉松彬;楊廣文;;CarrierFS:基于虛擬內(nèi)存的分布式文件系統(tǒng)[J];華中科技大學(xué)學(xué)報(bào)(自然科學(xué)版);2010年S1期
3 付印金;肖儂;劉芳;;重復(fù)數(shù)據(jù)刪除關(guān)鍵技術(shù)研究進(jìn)展[J];計(jì)算機(jī)研究與發(fā)展;2012年01期
4 方世昌;;國(guó)際標(biāo)準(zhǔn)ISO 7498-2第一版簡(jiǎn)介和讀后感[J];計(jì)算機(jī)工程與應(yīng)用;1990年07期
5 張前進(jìn);齊美彬;李莉;;基于應(yīng)用層負(fù)載均衡策略的分析與研究[J];計(jì)算機(jī)工程與應(yīng)用;2007年32期
6 楊德志;許魯;張建剛;;藍(lán)鯨分布式文件系統(tǒng)元數(shù)據(jù)服務(wù)[J];計(jì)算機(jī)工程;2008年07期
7 黎哲,郭成城,陳亮;一個(gè)基于TCP遷移機(jī)制的第七層負(fù)載均衡系統(tǒng)[J];計(jì)算機(jī)應(yīng)用研究;2005年04期
8 羅擁軍;李曉樂(lè);孫如祥;;負(fù)載均衡算法綜述[J];科技情報(bào)開(kāi)發(fā)與經(jīng)濟(jì);2008年23期
9 謝鯤;文吉?jiǎng)?張大方;謝高崗;;布魯姆過(guò)濾器查詢(xún)算法[J];軟件學(xué)報(bào);2009年01期
10 譚生龍;;存儲(chǔ)虛擬化技術(shù)的研究[J];微計(jì)算機(jī)應(yīng)用;2010年01期
相關(guān)碩士學(xué)位論文 前1條
1 陳虎;基于HDFS的云存儲(chǔ)平臺(tái)的優(yōu)化與實(shí)現(xiàn)[D];華南理工大學(xué);2012年
，本文編號(hào)：2319262
本文鏈接：http://sikaile.net/kejilunwen/jisuanjikexuelunwen/2319262.html
最近更新
教材專(zhuān)著
