本文選題：中文分詞　切入點(diǎn)：農(nóng)業(yè)垂直搜索引擎　出處：《新疆農(nóng)業(yè)大學(xué)》2013年碩士論文

【摘要】：本文首先對(duì)現(xiàn)有中文分詞相關(guān)理論和方法以及存在的主要問題進(jìn)行了深入分析，重點(diǎn)研究了統(tǒng)計(jì)模型在自然語(yǔ)言處理領(lǐng)域中的應(yīng)用。在此基礎(chǔ)上針對(duì)農(nóng)業(yè)垂直搜索領(lǐng)域的特殊需求及應(yīng)用環(huán)境，提出了基于詞典和統(tǒng)計(jì)語(yǔ)言模型的中文分詞方法。該方法通過改進(jìn)的全切分算法建立分詞矩陣，實(shí)現(xiàn)了所有類型歧義的識(shí)別，生成粗分結(jié)果集，然后利用N元語(yǔ)法模型從中選擇概率最大的切分結(jié)果，經(jīng)過基于最大熵模型的詞位標(biāo)注方法識(shí)別未登錄詞后得出最終的分詞結(jié)果，最后給出了基于此方法的中文分詞原型系統(tǒng)的設(shè)計(jì)與實(shí)現(xiàn)。 本文提出的分詞方法在三個(gè)方面進(jìn)行了改進(jìn)，首先是通過大規(guī)模語(yǔ)料庫(kù)識(shí)別具有切分標(biāo)志意義的特征字建立特征字庫(kù)，通過特征字對(duì)經(jīng)過預(yù)處理的語(yǔ)句集進(jìn)行初步切分，有效降低了粗分階段的字串長(zhǎng)度。其次采用改進(jìn)的全切分模型，通過字位標(biāo)注建立分詞矩陣，能夠有效檢測(cè)歧義邊界，識(shí)別所有類型的歧義，并篩選出包含歧義的切分形式，通過bigram模型進(jìn)行概率計(jì)算，選擇最優(yōu)切分形式。最后，建立農(nóng)業(yè)專業(yè)術(shù)語(yǔ)、中文人名、中文機(jī)構(gòu)名專業(yè)詞庫(kù)，統(tǒng)計(jì)構(gòu)詞規(guī)律，選擇合適的特征模板，生成樣本數(shù)據(jù)，利用詞位標(biāo)注的思想，通過最大熵模型實(shí)現(xiàn)對(duì)未登錄詞的識(shí)別。 本文設(shè)計(jì)了三個(gè)方面的實(shí)驗(yàn)，，對(duì)改進(jìn)的全切分算法和傳統(tǒng)全切分算法的性能進(jìn)行比較；在4詞位標(biāo)注集上選擇不同的上下文窗口寬度對(duì)基于最大熵模型的未登錄詞識(shí)別率的比較；將該原型系統(tǒng)與ICTCLAS、Paoding以及IKAnalyzer進(jìn)行綜合性能的比較。實(shí)驗(yàn)結(jié)果表明，使用本文提出的分詞模型的原型系統(tǒng)召回率達(dá)到93.6%，準(zhǔn)確率達(dá)到91.7%，F(xiàn)1測(cè)度值為92.6%，未登錄詞的召回率為77.2%，未登錄詞準(zhǔn)確率為90.1%。
[Abstract]:In this paper, the existing theories and methods of Chinese word segmentation and the main problems are analyzed. This paper focuses on the application of statistical model in the field of natural language processing. On this basis, it aims at the special needs and application environment in the field of vertical agricultural search. This paper proposes a Chinese word segmentation method based on dictionary and statistical language model, which establishes the segmentation matrix through the improved total segmentation algorithm, realizes the recognition of all types of ambiguity, and generates the rough result set. Then we use N-meta grammar model to select the segmentation result with the greatest probability, and get the final segmentation result after recognizing the unregistered words by the word location tagging method based on the maximum entropy model. Finally, the design and implementation of Chinese word segmentation prototype system based on this method are presented. The method of word segmentation proposed in this paper is improved in three aspects. Firstly, the feature database is established by large scale corpus recognition of feature words with the meaning of segmentation markers, and the pre-processed sentence set is segmented by feature words. The length of string in coarse stages is reduced effectively. Secondly, an improved total segmentation model is adopted, and word segmentation matrix is established by word tagging, which can effectively detect ambiguity boundaries, identify all types of ambiguity, and screen out segmentation forms that contain ambiguity. The bigram model is used to calculate the probability and select the optimal segmentation form. Finally, the specialized lexicon of agricultural terms, Chinese names, Chinese institutional names, statistical word-formation rules are established, and appropriate feature templates are selected to generate sample data. Using the idea of tagging words, the maximum entropy model is used to realize the recognition of unrecorded words. In this paper, three experiments are designed to compare the performance of the improved total segmentation algorithm and the traditional total segmentation algorithm. Selecting different context window width on 4-word tagging set to compare the recognition rate of unrecorded words based on maximum entropy model, and comparing the performance of the prototype system with ICTCLASS-Paoding and IKAnalyzer. The experimental results show that, The prototype system using the participle model proposed in this paper has a recall rate of 93.6, an accuracy of 91.7 / F _ 1 and a value of 92.6, a recall rate of 77.2 for unrecorded words and a accuracy of 90.1 for unrecorded words.
【學(xué)位授予單位】：新疆農(nóng)業(yè)大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2013
【分類號(hào)】：TP391.1

【參考文獻(xiàn)】

中國(guó)期刊全文數(shù)據(jù)庫(kù) 前10條

1 張文慧;張冉;;基于中文分詞的農(nóng)業(yè)信息檢索平臺(tái)設(shè)計(jì)[J];安徽農(nóng)業(yè)科學(xué);2011年20期

2 于江德;王希杰;;詞位標(biāo)注漢語(yǔ)分詞技術(shù)詳解[J];安陽(yáng)師范學(xué)院學(xué)報(bào);2010年05期

3 張德鑫;“水至清則無魚”——我的新生詞語(yǔ)規(guī)范觀[J];北京大學(xué)學(xué)報(bào)(哲學(xué)社會(huì)科學(xué)版);2000年05期

4 張鋒,樊孝忠;基于最大熵模型的交集型切分歧義消解[J];北京理工大學(xué)學(xué)報(bào);2005年07期

5 曹月雷;紀(jì)文彥;賈斌;;詞典與后綴數(shù)組相結(jié)合的中文分詞方法[J];硅谷;2012年21期

6 劉群,張華平,俞鴻魁,程學(xué)旗;基于層疊隱馬模型的漢語(yǔ)詞法分析[J];計(jì)算機(jī)研究與發(fā)展;2004年08期

7 劉遷;賈惠波;;中文信息處理中自動(dòng)分詞技術(shù)的研究與展望[J];計(jì)算機(jī)工程與應(yīng)用;2006年03期

8 曹波;蘇一丹;鄧琦;;基于最大熵模型的中國(guó)人名自動(dòng)識(shí)別[J];計(jì)算機(jī)工程與應(yīng)用;2009年04期

9 李國(guó)和;劉光勝;秦波波;吳衛(wèi)江;李洪奇;;綜合最大匹配和歧義檢測(cè)的中文分詞粗分方法[J];計(jì)算機(jī)工程與應(yīng)用;2012年14期

10 張仰森;;基于最大熵模型的漢語(yǔ)詞義消歧與標(biāo)注方法[J];計(jì)算機(jī)工程;2009年18期

中國(guó)重要會(huì)議論文全文數(shù)據(jù)庫(kù) 前1條

1 黃昌寧;趙海;;由字構(gòu)詞——中文分詞新方法[A];中文信息處理前沿進(jìn)展——中國(guó)中文信息學(xué)會(huì)二十五周年學(xué)術(shù)會(huì)議論文集[C];2006年

本文編號(hào)：1657432