本文選題：Web信息抽取技術　切入點：網頁模板　出處：《華東師范大學》2009年碩士論文

【摘要】： 隨著Internet的迅速發(fā)展,互聯(lián)網已成為一個巨大的信息庫,為了有效地利用互聯(lián)網上的信息,各種Web信息抽取技術應運而生。目前,Web上的很多網頁是網站根據用戶請求從后臺數據庫中選取數據并嵌入到通用的模板中,并結合網站的特定需求而動態(tài)生成的,例如電子商務網站的商品描述網頁等。針對如何從這類由模板生成的網頁中自動地抽取出有效數據的問題,目前常用的經典方法有RoadRunner,EXALG等,其中RoadRunner的實現算法的時間復雜度呈指數級增長,其實用性不強;雖然EXALG方法對RoadRunner方法進行了有效的改進,但是仍然缺乏對網頁中可視化布局信息、標記屬性和字符串的相似度等重要特征的考慮。因此,本文針對上述這些問題研討了相關網頁模板檢測問題的形式化描述,結合該類網頁的結構特征,探討了一種新的模板檢測方法;并且利用檢測出的模板完成對相關實例網頁的數據自動抽取過程;最終將該基于有效模板檢測的網頁數據自動抽取算法應用于某電子商務網站的相關網頁的數據抽取過程中,即對某網站中的商品列表信息和商品詳細信息等重要數據實現了自動抽取的工作。與其他方法相比,該方法能夠適應于“列表頁面”和“詳細頁面”兩種類型的網頁,在該類網頁數據抽取的查全率和準確率方面有了較大的改進。 本文的主要內容和結構安排如下: 首先,介紹針對模板生成網頁的數據抽取方法的發(fā)展現狀以及相關技術,并闡述了論文的研究目標和工作內容。 其次,介紹了Web數據抽取過程中主流的網頁數據抽取技術,系統(tǒng)地剖析了目前廣泛采用的經典的網頁數據抽取技術中存在的優(yōu)勢與不足,以此為基礎,文中研討了一種有效的針對模板生成網頁的數據抽取方法及其實現算法,即針對該類網頁,完成了相應網頁有效數據的自動抽取工作。 接著,重點闡述了文中所研討的針對模板生成網頁的數據自動抽取算法的設計與實現過程。該算法首先將已經凈化的HTML頁面解析成標簽樹和標簽隊列兩種數據結構;其次針對大部分網頁中存在導航條、廣告及版本信息等一些與抽取內容無關的數據信息,采用文中所提出的具體有效的標簽樹匹配算法過濾上述無關/冗余的數據信息;然后通過該數據自動抽取算法中計算Ctokens的核心子算法將這類HTML頁面進行有效的標簽歸類,以期基于所生成的Ctokens來自動抽取出該類網頁的模板結構信息數據和字段層次上的有效網頁生成數據。 最后,根據文中所研討的方法及實現算法,嘗試性地構造了一個針對模板生成網頁的數據自動抽取原型系統(tǒng),該系統(tǒng)能夠完成對相關電子商務網站中該類網頁(如:商品的“列表頁面”和“詳細頁面”的具體網頁)的有效數據的自動抽裙ぷ?該抽取過程的查全率和準確率都有較大的改進,所完成的工作是具有廣泛實際需求和深入推廣應用價值的。
[Abstract]:With the rapid development of Internet, the Internet has become a huge information base, in order to effectively use the information on the Internet, Web information extraction technology came into being. At present, a lot of Web pages is the site according to the request of the user selected data and embedded into the general template from databases, and websites with specific needs dynamically generated, such as electronic commerce website ". According to the description of the goods from the template generated web pages automatically extract the valid data, the classical methods of RoadRunner, EXALG and RoadRunner, which realized the time complexity of the algorithm grows exponentially, in fact is not strong; although the EXALG method the RoadRunner method is improved effectively, but there is still a lack of information visualization in web page layout, tag attributes and string similarity Other important features are considered. Therefore, aiming at these issues related web page template detection problem is formalized, combined with the structure characteristics of the web page, and discusses a new template detection method; and use the detected templates to complete automatic data extraction process of relevant examples of Web data extraction process; the web application based on web data template detection algorithm effective automatic extraction in an e-commerce site in the list of goods and merchandise information with information and other important data on a web site in the automatic extraction work. Compared with other methods, this method can be applied to the list of "pages" and "detail page" two types ", have been greatly improved in the aspect of the web data extraction recall and accuracy.
The main contents and structure of this paper are as follows:
First of all, this paper introduces the development status of data extraction method for template generation of web pages and related technologies, and expounds the research objectives and work content of the paper.
Secondly, introduces the web data extraction technology of Web data extraction process, systematically analyzes the existing web data extraction technology is widely used in the classical advantages and disadvantages, on this basis, this paper presents an effective template generated web pages data extraction method and algorithm for the class ", completed the work to automatically extract the corresponding page valid data.
Then, focuses on the design and implementation of template generated web pages automatic data extraction algorithm research in this paper. Firstly, HTML parsor had purified into two kinds of label label tree and queue data structure; secondly, there is a majority of web page navigation, independent advertising and version information and some the contents of the selected data, using the specific effective label tree is proposed in this paper, the algorithm of filtering irrelevant / redundant information; and then through the data extraction algorithm in computing core algorithm Ctokens the HTML page for effective label classification, which based on the generated Ctokens to automatically extract the data generated effective web template structure information of the data of the web page and the field level.
Finally, according to the studied method and algorithm of this paper attempts to construct a template generated web pages automatic data extraction prototype system, the system can complete the related e-commerce website in the web page (such as: "specific products" list "and" detail page ") automatic extraction the work of the skirt of the effective data extraction process? The recall and precision are greatly improved, the completion of the work has wide actual demand and thorough promotion application value.

【學位授予單位】：華東師范大學
【學位級別】：碩士
【學位授予年份】：2009
【分類號】：TP393.092

【參考文獻】

相關期刊論文 前2條

1 陳少飛,郝亞南,李天柱,徐林昊,楊文柱;Web信息抽取技術研究進展[J];河北大學學報(自然科學版);2003年01期

2 李保利,陳玉忠,俞士汶;信息抽取研究綜述[J];計算機工程與應用;2003年10期

，

本文編號：1676627