【摘要】：近年來(lái),信息產(chǎn)業(yè)在各行各業(yè)的飛速發(fā)展催生了行業(yè)數(shù)據(jù)的爆炸性增長(zhǎng),當(dāng)然也包括教育高考領(lǐng)域。眾所周知,每年的高考都會(huì)產(chǎn)生海量的高考數(shù)據(jù),如何快速高效的存儲(chǔ)這些海量的高考數(shù)據(jù)是一個(gè)值得研究的重要課題。在面對(duì)TB級(jí)別甚至是PB級(jí)別的海量數(shù)據(jù)時(shí)傳統(tǒng)的關(guān)系型數(shù)據(jù)庫(kù)對(duì)數(shù)據(jù)的存儲(chǔ)能力日漸乏力。伴隨著大規(guī)模數(shù)據(jù)的出現(xiàn)催生了不少存儲(chǔ)這些數(shù)據(jù)的技術(shù)。其中谷歌的GFS和Apache的HDFS就是兩種比較典型的大數(shù)據(jù)分布式存儲(chǔ)技術(shù),而在當(dāng)下比較受歡迎的當(dāng)屬Apache公司的HDFS。HDFS的出現(xiàn)使得企業(yè)可以采用大量廉價(jià)機(jī)器組成的集群對(duì)海量的數(shù)據(jù)進(jìn)行分布式存儲(chǔ),但是HDFS的分布式文件存儲(chǔ)由一個(gè)主節(jié)點(diǎn)控制多個(gè)從數(shù)據(jù)節(jié)點(diǎn)的存儲(chǔ)方式容易出現(xiàn)主節(jié)點(diǎn)瓶頸問(wèn)題。對(duì)于本文研究的高考數(shù)據(jù)來(lái)說(shuō),如果采用HDFS來(lái)存儲(chǔ)海量的高考數(shù)據(jù),當(dāng)大量的考生同時(shí)在線(xiàn)查詢(xún)成績(jī)時(shí),來(lái)自不同客戶(hù)端的請(qǐng)求都會(huì)涌入HDFS的主節(jié)點(diǎn),這對(duì)HDFS的主節(jié)點(diǎn)來(lái)說(shuō)是一個(gè)極大的挑戰(zhàn)。針對(duì)上述問(wèn)題,本文通過(guò)對(duì)HDFS分布式存儲(chǔ)技術(shù)進(jìn)行深入研究分析,提出了一種HDFS+MongoDB的分布式存儲(chǔ)方案來(lái)解決HDFS主節(jié)點(diǎn)瓶頸問(wèn)題,從而使高考數(shù)據(jù)分布式存儲(chǔ)更加優(yōu)化,考生查詢(xún)成績(jī)效率更高�；谏鲜龇治�,本文主要的研究工作如下:(1)首先明確了課題的選題背景及意義,接著對(duì)論文中應(yīng)用到分布式存儲(chǔ)技術(shù)、高考信息化技術(shù)、以及Spark大數(shù)據(jù)平臺(tái)技術(shù)的發(fā)展現(xiàn)狀進(jìn)行了分析。(2)分析了利用HDFS分布式存儲(chǔ)技術(shù)存儲(chǔ)高考數(shù)據(jù)出現(xiàn)的主節(jié)點(diǎn)瓶頸問(wèn)題,進(jìn)而提出了一種HDFS+MongoDB的分布式存儲(chǔ)高考數(shù)據(jù)的優(yōu)化方案。(3)根據(jù)招生考試院的具體要求對(duì)采用優(yōu)化存儲(chǔ)方案的高考成績(jī)查詢(xún)系統(tǒng)從用戶(hù)角度、功能性角度、非功能性角度等進(jìn)行了詳細(xì)的需求分析,并根據(jù)需求分析從系統(tǒng)的總體結(jié)構(gòu)、系統(tǒng)功能、系統(tǒng)數(shù)據(jù)庫(kù)和HDFS+MongoDB分布式存儲(chǔ)等四個(gè)方面進(jìn)行了詳細(xì)設(shè)計(jì)。(4)基于系統(tǒng)的詳細(xì)設(shè)計(jì)給出了具體的實(shí)現(xiàn)方法。采用黑盒測(cè)試方法對(duì)系統(tǒng)的功能進(jìn)行了測(cè)試,并從響應(yīng)時(shí)間、吞吐量、并發(fā)量三個(gè)方面對(duì)系統(tǒng)進(jìn)行了性能測(cè)試,初步達(dá)到了預(yù)期目標(biāo)。最后簡(jiǎn)單闡述了本文主要的研究?jī)?nèi)容,并明確了接下來(lái)要努力的方向。
[Abstract]:In recent years, the rapid development of information industry in various industries has given birth to the explosive growth of industry data, including, of course, the field of college entrance examination. As we all know, the college entrance examination every year will produce a huge amount of college entrance examination data, how to store these large amounts of college entrance examination data quickly and efficiently is an important topic worth studying. In the face of TB level or even PB level of massive data, the traditional relational database data storage capacity is increasingly weak. With the emergence of large-scale data, the emergence of a lot of data storage technology. Among them, Google's GFS and Apache's HDFS are two typical big data distributed storage technologies. The emergence of HDFS.HDFS, which is now a popular Apache company, allows enterprises to use clusters of cheap machines to store large amounts of data in a distributed manner. But the distributed file storage of HDFS is controlled by one master node, and the storage mode of multiple slave data nodes is prone to the bottleneck problem of master node. For the college entrance examination data studied in this paper, if we use HDFS to store a large amount of college entrance examination data, when a large number of candidates simultaneously online query results, the requests from different clients will flood into the main node of HDFS. This is a great challenge for the master node of HDFS. In view of the above problems, through the in-depth study and analysis of HDFS distributed storage technology, this paper proposes a distributed storage scheme of HDFS MongoDB to solve the bottleneck problem of HDFS master node, thus making the distributed storage of college entrance examination data more optimized. Examinee inquiry results are more efficient. Based on the above analysis, the main research work of this paper is as follows: (1) firstly, the background and significance of the topic are defined, and then the distributed storage technology, the college entrance examination information technology, are applied to the thesis. And the development of Spark big data platform technology is analyzed. (2) the bottleneck problem of main node in storing college entrance examination data using HDFS distributed storage technology is analyzed. Then an optimization scheme of distributed storage of college entrance examination data based on HDFS MongoDB is proposed. (3) according to the specific requirements of the college entrance examination institute, the query system of college entrance examination results using the optimized storage scheme is from the user's point of view, functional point of view, According to the requirement analysis, the system structure, the system function and the function of the system are analyzed in detail from the point of view of non-functional. The system database and HDFS MongoDB distributed storage are designed in detail. (4) based on the detailed design of the system, the implementation method is given. The function of the system is tested by using the black box test method, and the performance of the system is tested from three aspects: response time, throughput and concurrency. Finally, the main research content of this paper is briefly described, and the direction of the next efforts is defined.
【學(xué)位授予單位】：山東師范大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2017
【分類(lèi)號(hào)】：TP333

【參考文獻(xiàn)】

相關(guān)期刊論文 前10條

1 邱麗娟;;大數(shù)據(jù)處理平臺(tái)Spark基礎(chǔ)實(shí)踐研究[J];無(wú)線(xiàn)互聯(lián)科技;2017年01期

2 蔡文濤;;Spark大數(shù)據(jù)處理平臺(tái)的構(gòu)建及應(yīng)用[J];中國(guó)新通信;2016年15期

3 王康;李東靜;陳海光;;分布式存儲(chǔ)系統(tǒng)中改進(jìn)的一致性哈希算法[J];計(jì)算機(jī)技術(shù)與發(fā)展;2016年07期

4 賀靜霞;;高考教育功能的異化及其回歸[J];中國(guó)教育學(xué)刊;2016年05期

5 Yang Liu;Feng Yang;;Scala tympani drill-out technique for oval window atresia with malformed facial nerve: A report of three cases[J];Journal of Otology;2015年04期

6 劉峰波;;大數(shù)據(jù)Spark技術(shù)研究[J];數(shù)字技術(shù)與應(yīng)用;2015年09期

7 陳豪;謝曉蘭;;基于云技術(shù)的校園服務(wù)系統(tǒng)服務(wù)器端設(shè)計(jì)研究[J];科技視界;2015年09期

8 郭昕;朱春暉;;從教育公平視角探討我國(guó)的高考制度改革[J];湖南科技大學(xué)學(xué)報(bào)(社會(huì)科學(xué)版);2014年02期

9 王月春;;基于HDFS的遠(yuǎn)程教育課件資源管理[J];網(wǎng)絡(luò)安全技術(shù)與應(yīng)用;2013年09期

10 李唯;;學(xué)生成績(jī)管理系統(tǒng)的設(shè)計(jì)與實(shí)現(xiàn)[J];軟件導(dǎo)刊;2012年12期

相關(guān)會(huì)議論文 前1條

1 田原;王營(yíng)康;肖達(dá);楊榆;;云存儲(chǔ)系統(tǒng)中的存儲(chǔ)與數(shù)據(jù)拆分方案[A];第十九屆全國(guó)青年通信學(xué)術(shù)年會(huì)論文集[C];2014年

相關(guān)重要報(bào)紙文章 前2條

1 嚴(yán)雪林;;中國(guó)企業(yè)大數(shù)據(jù)應(yīng)用現(xiàn)狀及趨勢(shì)[N];現(xiàn)代物流報(bào);2014年

2 別志銘;;基于云的大數(shù)據(jù)分析系統(tǒng)[N];網(wǎng)絡(luò)世界;2013年

相關(guān)碩士學(xué)位論文 前10條

1 崔鑫;海量空間數(shù)據(jù)的分布式存儲(chǔ)管理及并行處理技術(shù)研究[D];國(guó)防科學(xué)技術(shù)大學(xué);2010年

2 童明;基于HDFS的分布式存儲(chǔ)研究與應(yīng)用[D];華中科技大學(xué);2012年

3 郭匡宇;基于MongoDB的傳感器數(shù)據(jù)分布式存儲(chǔ)的研究與應(yīng)用[D];南京郵電大學(xué);2013年

4 段弘;基于Play的用戶(hù)匹配與內(nèi)容推薦系統(tǒng)設(shè)計(jì)與實(shí)現(xiàn)[D];電子科技大學(xué);2013年

5 李吉檀;基于教育公平視角的我國(guó)異地高考制度研究[D];武漢理工大學(xué);2013年

6 唐振坤;基于Spark的機(jī)器學(xué)習(xí)平臺(tái)設(shè)計(jì)與實(shí)現(xiàn)[D];廈門(mén)大學(xué);2014年

7 高正九;基于HDFS的云存儲(chǔ)系統(tǒng)的設(shè)計(jì)與實(shí)現(xiàn)[D];中國(guó)科學(xué)技術(shù)大學(xué);2014年

8 孔曉斌;面向高考招生的智能數(shù)據(jù)分析系統(tǒng)研究[D];太原科技大學(xué);2008年

9 呂林;基于MongoDB的應(yīng)用平臺(tái)的研究與實(shí)現(xiàn)[D];北京郵電大學(xué);2015年

10 李爽;基于Spark的數(shù)據(jù)處理分析系統(tǒng)的設(shè)計(jì)與實(shí)現(xiàn)[D];北京交通大學(xué);2015年

，

本文編號(hào)：2408652