本文選題：Hadoop + LDA主題模型��； 參考：《北京郵電大學(xué)》2015年碩士論文

【摘要】：隨著互聯(lián)網(wǎng)的極速發(fā)展,互聯(lián)網(wǎng)產(chǎn)生的信息數(shù)據(jù)成爆炸式增長�；ヂ�(lián)網(wǎng)已經(jīng)從信息匱乏迅速轉(zhuǎn)入信息過于龐大而難以甄選有效信息的時(shí)代。文本信息作為傳遞信息的一種載體,依然是人們從互聯(lián)網(wǎng)中獲得信息的一種主要方式。如何從互聯(lián)網(wǎng)海量文本信息中獲取感興趣的信息是大數(shù)據(jù)挖掘中的一項(xiàng)重要任務(wù)。文本信息挖掘被廣泛應(yīng)用于網(wǎng)絡(luò)輿情、營銷、商業(yè)推薦等各方面,研究文本信息挖掘技術(shù)具有廣闊的市場應(yīng)用前景。另外一方面,海量數(shù)據(jù)的涌現(xiàn),使得傳統(tǒng)的服務(wù)器已經(jīng)無法承載海量數(shù)據(jù)的存儲(chǔ)和運(yùn)算,分布式系統(tǒng)已經(jīng)成為當(dāng)前處理海量數(shù)據(jù)的主流平臺(tái)。因此,如何將傳統(tǒng)的串行數(shù)據(jù)處理方法有效應(yīng)用到分布式系統(tǒng)中也成為了分布式系統(tǒng)研究的一個(gè)主要問題。 本文根據(jù)海量文本信息挖掘的核心問題,研究了基于Hadoop平臺(tái)的文本聚類問題,以便于利用分布式平臺(tái)提高文本聚類的效率和擴(kuò)大文本聚類處理數(shù)據(jù)的容量。本文取得的主要成果有： 1.本文根據(jù)Hadoop平臺(tái)的特性,實(shí)現(xiàn)并改進(jìn)了基于Mapreduce架構(gòu)的分布式LDA主題模型并行算法,有效解決了單機(jī)LDA中處理數(shù)據(jù)容量受硬件資源限制的問題。實(shí)驗(yàn)結(jié)果表明,分布式LDA主題模型在處理海量數(shù)據(jù)時(shí)具有明顯的時(shí)間優(yōu)勢(shì)。 2.本文設(shè)計(jì)和實(shí)現(xiàn)了一個(gè)可視化的Hadoop集群管理平臺(tái),通過該平臺(tái)簡化了Hadoop集群的管理。同時(shí),平臺(tái)引入了用戶權(quán)限控制模塊,增強(qiáng)了平臺(tái)的安全性。 3.本文利用實(shí)驗(yàn)室的閑置計(jì)算機(jī)資源,搭建了由25臺(tái)普通PC機(jī)構(gòu)成的Hadoop的集群,并在該平臺(tái)上驗(yàn)證和測(cè)試了可視化管理平臺(tái)和并行LDA主題模型的算法。該系統(tǒng)可以穩(wěn)定可靠地運(yùn)行。
[Abstract]:With the rapid development of the Internet, the information generated by the Internet has exploded. The Internet has rapidly shifted from a lack of information to an era of information too large to select valid information. Text information, as a carrier of information transmission, is still a main way for people to obtain information from the Internet. How to obtain interesting information from the massive text information of Internet is an important task of big data mining. Text information mining is widely used in network public opinion, marketing, commercial recommendation and other aspects, research text information mining technology has a broad market application prospects. On the other hand, with the emergence of mass data, traditional servers can no longer carry the storage and operation of mass data, and distributed system has become the mainstream platform to deal with mass data. Therefore, how to effectively apply the traditional serial data processing methods to distributed systems has become a major problem in the research of distributed systems. According to the core problem of massive text information mining, the text clustering problem based on Hadoop platform is studied in this paper, in order to improve the efficiency of text clustering and expand the capacity of text clustering processing data by using distributed platform. The main achievements of this paper are as follows: 1. According to the characteristics of Hadoop platform, the parallel algorithm of distributed LDA topic model based on Mapreduce architecture is implemented and improved in this paper, which effectively solves the problem that the data processing capacity in single LDA is limited by hardware resources. The experimental results show that the distributed LDA topic model has obvious time advantage in dealing with massive data. 2. This paper designs and implements a visual Hadoop cluster management platform, which simplifies the management of Hadoop cluster. At the same time, the platform introduces the user rights control module to enhance the security of the platform. 3. Based on the idle computer resources of the laboratory, this paper sets up a cluster of Hadoop composed of 25 ordinary PCs, and verifies and tests the algorithms of the visual management platform and the parallel LDA subject model on the platform. The system can operate stably and reliably.
【學(xué)位授予單位】：北京郵電大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2015
【分類號(hào)】：TP391.1

【參考文獻(xiàn)】

相關(guān)期刊論文 前10條

1 王旭仁;姚葉鵬;冉春風(fēng);何發(fā)鎂;;一種并行LDA主題模型建立方法研究[J];北京理工大學(xué)學(xué)報(bào);2013年06期

2 唐永瑞;張達(dá)敏;;基于Ajax與MVC模式的信息系統(tǒng)的研究與設(shè)計(jì)[J];電子技術(shù)應(yīng)用;2014年02期

3 和璇;;MVC模式在JAVA WEB中的應(yīng)用[J];計(jì)算機(jī)光盤軟件與應(yīng)用;2014年16期

4 李文波;孫樂;張大鯤;;基于Labeled-LDA模型的文本分類新算法[J];計(jì)算機(jī)學(xué)報(bào);2008年04期

5 徐戈;王厚峰;;自然語言處理中主題模型的發(fā)展[J];計(jì)算機(jī)學(xué)報(bào);2011年08期

6 嚴(yán)璋鵬;;基于B-S的學(xué)生學(xué)籍管理系統(tǒng)的設(shè)計(jì)與實(shí)現(xiàn)[J];價(jià)值工程;2013年19期

7 姜曉偉;王建民;丁貴廣;;基于主題模型的微博重要話題發(fā)現(xiàn)與排序方法[J];計(jì)算機(jī)研究與發(fā)展;2013年S1期

8 董新華;李瑞軒;周灣灣;王聰;薛正元;廖東杰;;Hadoop系統(tǒng)性能優(yōu)化與功能增強(qiáng)綜述[J];計(jì)算機(jī)研究與發(fā)展;2013年S2期

9 歐衛(wèi);謝贊福;謝彬彬;歐繽憶;;基于LDA模型的社交網(wǎng)絡(luò)主題社區(qū)挖掘[J];計(jì)算機(jī)與現(xiàn)代化;2014年08期

10 許伯熹;胡寧;陳文斌;高衛(wèi)國;程晉;;LDA算法在Mahout下的高效實(shí)現(xiàn)(英文)[J];華東師范大學(xué)學(xué)報(bào)(自然科學(xué)版);2013年03期

，

本文編號(hào)：1825336