【摘要】：隨著網(wǎng)絡(luò)的迅速發(fā)展,萬維網(wǎng)成為大量信息的載體,如何有效地提取并利用這些信息成為一個(gè)巨大的挑戰(zhàn)。為滿足這種需求,網(wǎng)絡(luò)爬蟲應(yīng)運(yùn)而生。它是一種按照一定的規(guī)則,自動(dòng)的抓取萬維網(wǎng)信息的程序或者腳本。 首先,本文介紹了網(wǎng)絡(luò)爬蟲的發(fā)展歷史及其應(yīng)用領(lǐng)域,通過分析主流的網(wǎng)絡(luò)爬蟲,發(fā)現(xiàn)當(dāng)今的網(wǎng)絡(luò)爬蟲主要服務(wù)于搜索引擎,為面向主題的用戶查詢準(zhǔn)備數(shù)據(jù)資源。從網(wǎng)絡(luò)爬蟲的拓展性極強(qiáng)的爬行架構(gòu)出發(fā),傳統(tǒng)爬蟲對于搜索引擎的重要性逐漸淡化了它在靈活性和功能性上的特點(diǎn)。然后,本文探討了評價(jià)網(wǎng)路爬蟲性能的若干指標(biāo),并以此出發(fā),從性能提升和功能拓展兩個(gè)方面詳細(xì)介紹了中小型網(wǎng)絡(luò)爬蟲的優(yōu)化策略。 在性能提升方面,本文根據(jù)不同的功能模塊分別介紹了若干優(yōu)化方案。第一,選擇Gzip/deflate壓縮編碼傳輸,通過降低傳輸量來降低網(wǎng)絡(luò)傳輸時(shí)間；第二,異步請求下載,提高帶寬占用率和CPU利用率；第三,采用廣度優(yōu)先爬行,使用布隆過濾器實(shí)現(xiàn)大規(guī)模的URL去重檢測；第四,采用設(shè)計(jì)精細(xì)的正則表達(dá)式提取頁面鏈接；第五,對爬取的URL進(jìn)行嚴(yán)格的正規(guī)化處理,降低URL的出錯(cuò)對爬蟲的誤導(dǎo)性；第六,委托優(yōu)化后的線程池高效率地管理多線程。 在功能拓展方面,本文主要從以下三方面進(jìn)行區(qū)別于傳統(tǒng)爬蟲的新嘗試。第一,靜態(tài)頁面性能分析,給網(wǎng)站提供性能改進(jìn)意見；第二,充當(dāng)自動(dòng)化測試工具,用于在指定頁面執(zhí)行測試用例；第三,可定制的聚焦數(shù)據(jù)提取,根據(jù)用戶的需求進(jìn)行指定格式的數(shù)據(jù)抓取。 基于驗(yàn)證上述優(yōu)化策略的目的,.NET平臺(tái)特別適合架構(gòu)輕量級的爬蟲。該爬蟲采用.NET平臺(tái)下的C#語言,在Visual Studio2008環(huán)境下進(jìn)行開發(fā)。程序在命令行模式下運(yùn)行,具有基于文件的高可配置性。
[Abstract]:With the rapid development of the network, the World wide Web has become the carrier of a lot of information. How to extract and utilize the information effectively becomes a huge challenge. In order to meet this demand, the network crawler came into being. It is a program or script that automatically grabs World wide Web information according to certain rules. Firstly, this paper introduces the history of web crawler and its application field. By analyzing the mainstream web crawler, it is found that today's web crawler mainly serves search engine and prepares data resources for subject oriented user query. Based on the highly extensible crawling architecture of web crawlers, the importance of traditional crawlers to search engines has gradually weakened its flexibility and functional characteristics. Then, this paper discusses some indexes to evaluate the performance of web crawlers, and then introduces the optimization strategies of small and medium-sized web crawlers from two aspects of performance improvement and function expansion. In terms of performance improvement, this paper introduces several optimization schemes according to different function modules. First, choose Gzip-deflate compression code transmission to reduce the network transmission time by reducing the amount of transmission; second, asynchronous request download, increase bandwidth utilization and CPU utilization; third, use breadth first crawling, Using Bloom filter to achieve large-scale URL re-detection; fourth, using well-designed regular expressions to extract page links; fifthly, strictly regularizing the URL crawled to reduce the error of URL to the reptile misleading; sixth, The optimized thread pool efficiently manages multithreading. In the aspect of function expansion, this paper mainly tries to distinguish the traditional reptile from the following three aspects. First, static page performance analysis provides performance improvement advice to the website; second, it acts as an automated test tool for performing test cases on a specified page; third, customizable focused data extraction, According to the needs of the user for the specified format of data capture. Based on the verification of the above optimization strategy, the .NET platform is particularly suitable for lightweight crawlers. The crawler is developed in Visual Studio 2008 with C # language based on. Net platform. The program runs in command-line mode and is highly configurable based on files.
【學(xué)位授予單位】：吉林大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2012
【分類號(hào)】：TP391.3

【參考文獻(xiàn)】

相關(guān)期刊論文 前10條

1 李悅;;搜索引擎技術(shù)的產(chǎn)生與發(fā)展綜述[J];福建電腦;2010年05期

2 呂曉峰,董守斌,張凌;并行數(shù)據(jù)采集器任務(wù)分配策略的設(shè)計(jì)與實(shí)現(xiàn)[J];華中科技大學(xué)學(xué)報(bào)(自然科學(xué)版);2003年S1期

3 周源遠(yuǎn),王繼成,鄭剛,張福炎;Web頁面清洗技術(shù)的研究與實(shí)現(xiàn)[J];計(jì)算機(jī)工程;2002年09期

4 周立柱,林玲;聚焦爬蟲技術(shù)研究綜述[J];計(jì)算機(jī)應(yīng)用;2005年09期

5 尹江;尹治本;黃洪;;網(wǎng)絡(luò)爬蟲效率瓶頸的分析與解決方案[J];計(jì)算機(jī)應(yīng)用;2008年05期

6 王華,馬亮,顧明;線程池技術(shù)研究與應(yīng)用[J];計(jì)算機(jī)應(yīng)用研究;2005年11期

7 劉金紅;陸余良;;主題網(wǎng)絡(luò)爬蟲研究綜述[J];計(jì)算機(jī)應(yīng)用研究;2007年10期

8 程嵐嵐;;基于正則表達(dá)式的大規(guī)模網(wǎng)頁術(shù)語對抽取研究[J];情報(bào)雜志;2008年11期

9 許笑;張偉哲;張宏莉;方濱興;;廣域網(wǎng)分布式Web爬蟲[J];軟件學(xué)報(bào);2010年05期

10 鄒志華;陳玉健;劉強(qiáng);;一種維護(hù)WAP網(wǎng)站的網(wǎng)絡(luò)爬蟲的設(shè)計(jì)[J];微計(jì)算機(jī)信息;2006年21期

相關(guān)會(huì)議論文 前1條

1 樸星海;趙鐵軍;鄭德權(quán);張迪;;面向Blog的網(wǎng)絡(luò)爬行器設(shè)計(jì)與實(shí)現(xiàn)[A];中文信息處理前沿進(jìn)展——中國中文信息學(xué)會(huì)二十五周年學(xué)術(shù)會(huì)議論文集[C];2006年

相關(guān)碩士學(xué)位論文 前3條

1 何世林;基于Java技術(shù)的搜索引擎研究與實(shí)現(xiàn)[D];西南交通大學(xué);2006年

2 朱良峰;主題網(wǎng)絡(luò)爬蟲的研究與設(shè)計(jì)[D];南京理工大學(xué);2008年

3 劉喜亮;面向主題的網(wǎng)絡(luò)爬蟲設(shè)計(jì)與實(shí)現(xiàn)[D];湖南大學(xué);2009年

，

本文編號(hào)：2137449