本文選題：文檔部分重復檢測 + Low-IDF-SIG特征提取算法��； 參考：《復旦大學》2012年碩士論文

【摘要】：隨著互聯網上數據的爆炸式的增長,互聯網上產生了大量的重復數據。這些重復數據給搜索引擎、觀點挖掘等許多Web應用帶來了嚴峻的問題。目前絕大部分的重復檢測的算法均著重考慮文檔級別,這些方法不能有效地檢測出兩個文檔中只有一部分互為重復的情況。 本文提出了一種算法以解決文檔部分重復檢測問題。該方法分為句子級別的重復檢測以及序列匹配兩個子問題。首先,本文提出了一種快速有效的句子級別的特征提取方法—Low-IDF-SIG算法,并基于該算法實現了一個可以高效地找出句子級別重復的檢測系統(tǒng)。為了對本文提出的方法的精度及效率進行評測,作者還在一個真實的語料庫上對提出的方法與其他方法進行了比較。實驗結果證明本文提出的方法能有效地提高句子級別的重復檢測任務的效率和精度。 此外本文還提出了基于MapReduce范式的文檔部分重復檢測算法PDC-MR-Ⅱ算法。并基于該算法實現了一個基于MapReduce范式的高效的分布式文檔部分重復檢測系統(tǒng)。 本文中提出的算法和實現的系統(tǒng)可以廣泛用于解決論文抄襲檢測,論壇話題重復檢測、分頁新聞的重復檢測等課題。
[Abstract]:With the explosive growth of data on the Internet, a large number of duplicate data have been generated on the Internet.These repeated data bring severe problems to many Web applications such as search engine, viewpoint mining and so on.At present, most of the repeated detection algorithms focus on the document level, and these methods can not effectively detect the situation that only one part of the two documents is duplicated with each other.This paper presents an algorithm to solve the problem of document partial repetition detection.The method is divided into two sub-problems: sentence level repetition detection and sequence matching.Firstly, this paper proposes a fast and effective sentence-level feature extraction method-Low-IDF-SIG algorithm, and implements a detection system based on this algorithm, which can efficiently find sentence level repetition.In order to evaluate the accuracy and efficiency of the proposed method, the author also compares the proposed method with other methods on a real corpus.The experimental results show that the proposed method can effectively improve the efficiency and accuracy of sentence level repeat detection.In addition, PDC-MR- 鈪，

本文編號：1760120