本文選題：網(wǎng)絡(luò)爬蟲 + 原子詞�。� 參考：《北京郵電大學(xué)》2012年碩士論文

【摘要】：從古至今,信息對于人類的生產(chǎn)生活等方方面面都是非常重要的,特別是對于處于信息時(shí)代的我們更是如此。隨著互聯(lián)網(wǎng)技術(shù)的飛速發(fā)展,互聯(lián)網(wǎng)己成為我們獲取各種信息的主要渠道,但是充斥于互聯(lián)網(wǎng)中的信息每天都在以指數(shù)規(guī)模增長且各種信息交織在一起,在這種情況下,如何準(zhǔn)確獲取有效信息自然成為了研究的重點(diǎn)。 聚類分析是自然語言處理技術(shù)中的一項(xiàng)重要技術(shù),是挖掘隱藏在海量信息背后的有效信息的重要方法。對于科學(xué)研究來說,一方面,各類論文、期刊、文獻(xiàn)數(shù)量龐大難以全部閱讀；另一方面,搜索引擎技術(shù)的大量運(yùn)用也提供了數(shù)量巨大的詞匯來幫助我們發(fā)現(xiàn)各種信息,因此如何通過對各種已有詞匯進(jìn)行聚類來發(fā)現(xiàn)有用信息成為了一個(gè)有實(shí)際意義的課題。 本文首先分析了在當(dāng)今信息爆炸的背景下,科學(xué)研究工作所面臨的困境——文獻(xiàn)檢索技術(shù)無法滿足實(shí)際應(yīng)用需求,接著對文獻(xiàn)檢索技術(shù)進(jìn)行了介紹,特別是對于其核心模塊之一的網(wǎng)絡(luò)爬蟲進(jìn)行了深入的研究,并對時(shí)下網(wǎng)絡(luò)爬蟲研究的重點(diǎn)方向——聚焦網(wǎng)絡(luò)爬蟲進(jìn)行了全面和細(xì)致的學(xué)習(xí),包括其體系結(jié)構(gòu)、關(guān)鍵技術(shù)等。其次,本文簡要介紹了自然語言處理中的聚類技術(shù),并在此基礎(chǔ)上分別介紹了詞聚類技術(shù)、概念聚類技術(shù)。通過對目前流行的詞聚類技術(shù)進(jìn)行仔細(xì)分析,本文針對其聚類空間維度通常過高導(dǎo)致聚類復(fù)雜度過高的問題采用了基于原子概念的詞聚類以期降低聚類復(fù)雜度,最終目的是結(jié)合網(wǎng)絡(luò)爬蟲技術(shù)與詞聚類技術(shù),通過在海量信息背景下基于原子概念的詞聚類來解決當(dāng)前由于信息爆炸導(dǎo)致的科學(xué)研究熱點(diǎn)發(fā)現(xiàn)困難的問題。最后,在深入研究了上述理論知識(shí)的基礎(chǔ)上,本文一方面設(shè)計(jì)并實(shí)現(xiàn)了用于從指定網(wǎng)站抓取指定數(shù)據(jù)的網(wǎng)絡(luò)爬蟲程序,另一方面,本文充分利用自然語言處理技術(shù)中的中文詞聚類技術(shù)借助MATLAB中的FCM算法實(shí)現(xiàn)了基于原子概念的中文詞聚類系統(tǒng),并對實(shí)驗(yàn)結(jié)果進(jìn)行了分析,基本取得了預(yù)期的效果。
[Abstract]:From ancient to present, information is very important for all aspects of human production and life, especially for us in the information age. With the rapid development of Internet technology, the Internet has become the main channel for us to obtain all kinds of information, but the information in the Internet is growing exponentially every day and all kinds of information are intertwined together, in this case, How to accurately obtain effective information has naturally become the focus of research. Clustering analysis is an important technology in natural language processing, and it is an important method to mine the effective information hidden behind massive information. For scientific research, on the one hand, the large number of papers, periodicals and documents is difficult to read; on the other hand, the extensive use of search engine technology also provides a large number of words to help us find all kinds of information. Therefore, how to find useful information by clustering all kinds of existing words has become a meaningful topic. This paper first analyzes the predicament of scientific research work under the background of information explosion, that is, the literature retrieval technology can not meet the practical application needs, and then introduces the literature retrieval technology. In particular, the network crawler, one of its core modules, has been deeply studied, and the focus of the current research on web crawler-focused web crawler, including its architecture, key technology and so on, has been studied comprehensively and meticulously. Secondly, this paper briefly introduces the clustering technology in natural language processing, and then introduces the word clustering technology and the concept clustering technology respectively. Through the careful analysis of the current popular word clustering techniques, this paper uses the word clustering based on the atomic concept to reduce the clustering complexity in order to reduce the clustering complexity, which is usually caused by the high spatial dimension of the clustering. The ultimate goal is to solve the problem of difficult scientific research hot spot discovery caused by information explosion through word clustering based on atomic concept in the context of massive information. The purpose of this paper is to combine the technology of web crawler and word clustering. Finally, based on the above theoretical knowledge, on the one hand, this paper designs and implements a web crawler program to capture the specified data from a designated website, on the other hand, In this paper, we make full use of the Chinese word clustering technology in natural language processing technology to realize the Chinese word clustering system based on atomic concept with the help of FCM algorithm in MATLAB. The experimental results are analyzed and the expected results are obtained.
【學(xué)位授予單位】：北京郵電大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2012
【分類號(hào)】：TP391.1;TP311.13

【參考文獻(xiàn)】

相關(guān)期刊論文 前10條

1 孫立偉;何國輝;吳禮發(fā);;網(wǎng)絡(luò)爬蟲技術(shù)的研究[J];電腦知識(shí)與技術(shù);2010年15期

2 戈國華;肖海波;張敏;;基于FCM的數(shù)據(jù)聚類分析及Matlab實(shí)現(xiàn)[J];福建電腦;2007年04期

3 袁里馳;;一種基于互信息的詞聚類算法[J];系統(tǒng)工程;2008年05期

4 紀(jì)曉華;;“聚焦爬蟲”技術(shù)[J];互聯(lián)網(wǎng)天地;2007年12期

5 吳亮;;搜索引擎中網(wǎng)絡(luò)爬蟲的設(shè)計(jì)[J];決策與信息(財(cái)經(jīng)觀察);2008年07期

6 周雅倩,郭以昆,黃萱菁,吳立德;基于最大熵方法的中英文基本名詞短語識(shí)別[J];計(jì)算機(jī)研究與發(fā)展;2003年03期

7 赫楓齡,左萬利;用有向圖法解決網(wǎng)頁爬行中循環(huán)鏈接問題[J];吉林大學(xué)學(xué)報(bào)(理學(xué)版);2004年03期

8 韓建超;王紅蕾;;機(jī)器學(xué)習(xí)中的概念聚類[J];計(jì)算機(jī)科學(xué);1991年05期

9 黃德才;戚華春;;PageRank算法研究[J];計(jì)算機(jī)工程;2006年04期

10 胡和平;曾慶銳;路松峰;;中文詞聚類研究[J];計(jì)算機(jī)工程與科學(xué);2006年01期

相關(guān)碩士學(xué)位論文 前4條

1 董建設(shè);基于HTML標(biāo)記分析及中文切詞的網(wǎng)頁索引研究與實(shí)現(xiàn)[D];蘭州理工大學(xué);2003年

2 張薇娟;基于模糊認(rèn)知圖的分步文本層次分類研究[D];天津師范大學(xué);2008年

3 劉喜亮;面向主題的網(wǎng)絡(luò)爬蟲設(shè)計(jì)與實(shí)現(xiàn)[D];湖南大學(xué);2009年

4 夏亮;主題搜索引擎網(wǎng)絡(luò)爬蟲搜索策略的研究與實(shí)現(xiàn)[D];北京化工大學(xué);2010年

，

本文編號(hào)：1787708