【摘要】：隨著互聯(lián)網(wǎng)的飛速發(fā)展,尤其是微博、在線廣告等應(yīng)用的涌現(xiàn),出現(xiàn)在互聯(lián)網(wǎng)上的短文本越來越多,對短文本的理解也就成為了一項非常重要的工作。大部分傳統(tǒng)文本挖掘方法都是針對長文本設(shè)計的。對于短文本,由于其表征的稀疏性,現(xiàn)有的大部分技術(shù)并不能有效地應(yīng)用在短文本上。為了更好地理解短文本,我們發(fā)現(xiàn)通�？梢哉业街黝}相關(guān)的長文本作為輔助數(shù)據(jù)來幫助短文本的理解。 本文描述了一個創(chuàng)新的短文本聚類方法,該方法通過從輔助長文本數(shù)據(jù)上遷移知識來幫助短文本聚類。大部分之前用來提高短文本聚類效果的相關(guān)工作忽略了短文本和輔助長文本之間的語義及主題不一致性,為了解決這些存在于目標(biāo)數(shù)據(jù)和輔助數(shù)據(jù)間的不一致性,我們提出了一種新的主題模型,二元隱含狄利克雷分配模型(DLDA)。該模型同時從長文本和短文本數(shù)據(jù)中學(xué)習(xí)主題,為了針對長短文本語言上的不一致性,我們設(shè)計了兩個模型來區(qū)別對待長短文本。一種模型通過調(diào)整文檔主題分布的先驗對文檔集的主題選擇進行控制,一種模型通過改進文檔生成過程的假設(shè)自動控制不同文檔對主題的選擇。 通過在廣告和微博(Twitter)數(shù)據(jù)上的大規(guī)模聚類實驗,證明我們的方法獲得了優(yōu)于當(dāng)今主流方法的短文本聚類效果。同時也證明了,考慮目標(biāo)短文本數(shù)據(jù)集與輔助長文本數(shù)據(jù)集之間的差異可以對提升短文本的聚類效果有很大幫助。
[Abstract]:With the rapid development of the Internet, especially the emergence of Weibo, online advertising and other applications, more and more short text books appear on the Internet, so understanding of short texts has become a very important work. Most of the traditional text mining methods are designed for long text. For short text, due to its sparse representation, most of the existing techniques can not be effectively applied to short text. In order to better understand the short text, we find that we can usually find long text related to the topic as auxiliary data to help the understanding of short text. This paper describes an innovative short text clustering method, which helps short text clustering by transferring knowledge from auxiliary long text data. Most of the previous work used to improve the clustering effect of short texts ignored the semantic and thematic inconsistency between short text and auxiliary long text, in order to solve the inconsistency between target data and auxiliary data. We propose a new topic model, binary implicit Dirichlet assignment model (DLDA). In order to deal with the language inconsistency of long and short texts, we designed two models to treat short and short texts differently. A model controls the topic selection of a document set by adjusting a priori distribution of document topics, and a model automatically controls the topic selection of different documents by improving the assumptions of the document generation process. Through a large scale clustering experiment on advertising and Weibo (Twitter) data, it is proved that our method is better than the current mainstream method in short text clustering. It is also proved that considering the difference between the target short text dataset and the auxiliary long text data set can greatly improve the clustering effect of short text.
【學(xué)位授予單位】：上海交通大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2012
【分類號】：TP391.1

【共引文獻(xiàn)】

相關(guān)期刊論文 前2條

1 ;Short text classification based on strong feature thesaurus[J];Journal of Zhejiang University-Science C(Computers & Electronics);2012年09期

2 田野;王文東;饒京海;王冠;郭亮;陳燦峰;馬建;;短信息的會話檢測及組織[J];軟件學(xué)報;2012年10期

相關(guān)博士學(xué)位論文 前2條

1 倪興良;問答系統(tǒng)中的短文本聚類研究與應(yīng)用[D];中國科學(xué)技術(shù)大學(xué);2011年

2 路遙;用戶交互式問答系統(tǒng)中問題推薦機制的研究[D];中國科學(xué)技術(shù)大學(xué);2012年

，

本文編號：2349207