發(fā)布時(shí)間：2018-03-09 14:47

  本文選題：信息檢索　切入點(diǎn)：網(wǎng)絡(luò)爬蟲　出處：《復(fù)旦大學(xué)》2013年碩士論文　論文類型：學(xué)位論文

【摘要】：隨著現(xiàn)代信息技術(shù)的飛速發(fā)展,互聯(lián)網(wǎng)絡(luò)上的信息量和類型正在發(fā)生爆炸性的增長。這為人們的日常生活、工作以及學(xué)習(xí)帶來了極大的便利。但是在信息量爆增的同時(shí)也帶來了新的問題。比如如何對(duì)這些海量的信息進(jìn)行統(tǒng)一的管理,如何將這些分散的資源建立索引,以及如何從海量的信息資源中準(zhǔn)確地獲取需要的信息等等。搜索引擎是解決這些問題的關(guān)鍵技術(shù),但是傳統(tǒng)的通用搜索引擎是對(duì)Web上的所有種類的信息都進(jìn)行搜集,并面向所有不同層次的用戶,這種想做的面面俱到的努力在海量信息面前變得越來越?jīng)]有突破性進(jìn)展。普通的用戶對(duì)信息的關(guān)注程度和寬度是比較集中的。所以面向特定領(lǐng)域和特定需求的專業(yè)搜索引擎的概念應(yīng)用而生。與傳統(tǒng)的通用搜索引擎所不同的地方是專業(yè)搜索引擎只會(huì)收集與某個(gè)主題相關(guān)的Web上的信息,在收集信息時(shí)并不是來者便收,而是通過分析判斷信息內(nèi)容是否與特定主題相關(guān),并只對(duì)相關(guān)的信息進(jìn)行進(jìn)一步處理。因此,專業(yè)搜索引擎無論在資源消耗,還是在查詢準(zhǔn)確度上都有了顯著的提高。本文的主要研究工作就是面向?qū)I(yè)搜索引擎,且以新聞為搜索主題。在研究過程中,通過對(duì)搜索引擎中關(guān)鍵技術(shù)進(jìn)行深入的理論的學(xué)習(xí)和實(shí)踐,進(jìn)一步加深對(duì)搜索引擎領(lǐng)域的了解。在本文中的新聞專業(yè)搜索引擎中,選擇新浪新聞網(wǎng)站作為網(wǎng)絡(luò)爬蟲的入口地址,對(duì)其進(jìn)行有針對(duì)性地收集新聞頁面。收集頁面的工作由專業(yè)的新聞網(wǎng)絡(luò)爬蟲完成,它從新聞首頁開始,提取出其中的新聞鏈接地址,并將這些鏈接地址存入到待爬取的隊(duì)列之中,通過三層的深度優(yōu)先搜索算法對(duì)Web網(wǎng)站進(jìn)行遍歷。之后,爬蟲還將對(duì)收集后的頁面進(jìn)行凈化處理和提取有效信息,最后由索引器建立搜索引擎中非常核心的數(shù)據(jù)：倒排索引。搜索引擎最終是要面向普通用戶的,所以,設(shè)計(jì)好一個(gè)用戶體驗(yàn)度好的查詢接口為用戶提供新聞查詢服務(wù)也是非常必須的任務(wù)。本文中詳細(xì)介紹了網(wǎng)絡(luò)爬蟲是設(shè)計(jì)和實(shí)現(xiàn),網(wǎng)頁的凈化和信息抽取以及索引庫的構(gòu)建。這些技術(shù)都是目前自然語言處理和人工智能方面的研究熱點(diǎn),通過對(duì)這些技術(shù)和理論的學(xué)習(xí),加深對(duì)專業(yè)的技能。本面向新聞內(nèi)容的搜索引擎從最簡單的技術(shù)著手,逐步實(shí)現(xiàn)了搜索引擎這一龐雜系統(tǒng)中的關(guān)鍵模塊,實(shí)驗(yàn)結(jié)果表明系統(tǒng)具有一定的準(zhǔn)確率,達(dá)到了良好的效果。
[Abstract]:With the rapid development of modern information technology, the amount and type of information on the Internet is increasing explosively. Work and study bring great convenience. But as the amount of information explodes, it also brings new problems. For example, how to manage these huge amounts of information uniformly, how to index these scattered resources, Search engine is the key technology to solve these problems, but the traditional universal search engine is to collect all kinds of information on Web. And for all the different levels of users, This kind of all-encompassing effort in the face of mass information has become less and less groundless. The average user's attention to the information and width is more concentrated. So specific to specific areas and specific needs. Different from traditional general-purpose search engines, professional search engines only collect information on Web that is relevant to a particular topic. When collecting information, it is not collected by the person who comes, but by analyzing and judging whether the content of the information is relevant to a particular topic, and only the relevant information is further processed. Therefore, the professional search engine, regardless of the resource consumption, The main research work of this paper is to face professional search engine, and take news as the search subject. In the process of research, Through the deep theoretical study and practice of the key technologies in the search engine, we can further deepen our understanding of the search engine field. In this paper, we select the Sina news website as the entry address of the web crawler in the news professional search engine. The collection of pages is done by a professional news web crawler, who starts with the first page of the news and extracts the address of the news link. These link addresses are stored in the queue to be crawled, and the Web site is traversed by a three-layer depth-first search algorithm. After that, the crawler will purify the collected pages and extract effective information. Finally, the indexer builds the very core data in the search engine: inverted index. The search engine is ultimately intended for ordinary users, so, It is also a very necessary task to design a good user experience query interface to provide news query service for users. This paper introduces the design and implementation of web crawler in detail. The purification and information extraction of web pages and the construction of index database. These technologies are the research hotspot in the field of natural language processing and artificial intelligence. Through the study of these technologies and theories, The search engine for news content has gradually realized the key module of the complex system from the simplest technology. The experimental results show that the system has a certain accuracy. Good results have been achieved.
【學(xué)位授予單位】：復(fù)旦大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2013
【分類號(hào)】：TP391.3

【參考文獻(xiàn)】

相關(guān)期刊論文 前4條

1 唐波;;網(wǎng)絡(luò)爬蟲的設(shè)計(jì)與實(shí)現(xiàn)[J];電腦知識(shí)與技術(shù);2009年11期

2 張保富;施化吉;馬素琴;;基于TFIDF文本特征加權(quán)方法的改進(jìn)研究[J];計(jì)算機(jī)應(yīng)用與軟件;2011年02期

3 俞平;肖南峰;甘志剛;;第三代搜索引擎研究[J];南京信息工程大學(xué)學(xué)報(bào)(自然科學(xué)版);2009年02期

4 楊思洛;搜索引擎的排序技術(shù)研究[J];現(xiàn)代圖書情報(bào)技術(shù);2005年01期

相關(guān)碩士學(xué)位論文 前1條

1 劉喜亮;面向主題的網(wǎng)絡(luò)爬蟲設(shè)計(jì)與實(shí)現(xiàn)[D];湖南大學(xué);2009年

，

本文編號(hào)：1588987