本文關鍵詞： 中文分詞 多模型 字標注 領域自適應 特征嵌入　出處：《沈陽航空航天大學》2017年碩士論文　論文類型：學位論文

【摘要】：中文分詞是指將連續(xù)的字序列依照特定的規(guī)范切分為合理的詞序列的過程。作為自然語言處理最基本的一個步驟,是信息檢索、知識獲取以及機器翻譯等應用必須處理的關鍵環(huán)節(jié)。因此,研究中文分詞具有重要的理論和現實意義。本文提出了一種基于字的多模型分詞方法。該方法采用神經網絡模型結構針對每個字單獨建立模型。由于中文漢字本身帶有語義信息,不同的字在不同語境中其含義和作用不同,造成每個字的構詞規(guī)律存在差異。與現有字標注分詞方法不同的是,該方法能夠有效區(qū)分每個特征對不同待切分字的影響,從而學習出字構詞的特殊性規(guī)律。通過與單模型方法、CRF方法以及前人的工作進行對比,本文提出的基于字的多模型方法取得了更好的分詞效果。并在SIGHAN Backoff2005提供的中文簡體語料PKU和MSR上,取得的F值分別為93.4%和95.5%。根據上述方法,面向領域自適應分詞任務,本文提出了一種基于字的領域自適應分詞方法。由于字模型相互獨立,模型更新時,保留遷移性能強的字模型,對遷移性能弱的字模型進行更新訓練。解決了大規(guī)模切分數據難與共享,源領域與目標領域數據混合需要重新訓練等問題。對目標領域進行分詞時,通過模型的自適應能力實現領域自適應。特征嵌入的表示方法能夠有效地解決特征稀疏問題,本文采用特征嵌入來表示輸入特征。實驗結果表明,本文提出的分詞方法能夠明顯提高領域適應性能力。最后,設計并實現了領域自適應中文分詞系統(tǒng)。該系統(tǒng)可以實現利用已有的基礎模型對輸入的句子或文本進行分詞,并且支持添加相關領域詞典,還可根據待分詞領域訓練數據對基礎模型進行更新,從而獲得相關領域較好的分詞結果。
[Abstract]:Chinese word segmentation refers to the process of dividing consecutive word sequences into rational word sequences according to specific norms. As the most basic step of natural language processing, information retrieval is one of the most important steps in Chinese word segmentation. Knowledge acquisition and machine translation and other applications must deal with the key links. It is of great theoretical and practical significance to study Chinese word segmentation. In this paper, a word-based multi-model word segmentation method is proposed. The neural network model structure is used to build a model for each word. Body with semantic information. Different words have different meanings and functions in different contexts, which result in different word-formation rules. This method can effectively distinguish the influence of each feature on the different words to be segmented, so as to learn the special rule of word-formation, and compare it with the single model method / CRF method and the previous work. The word-based multi-model method proposed in this paper has achieved better word segmentation effect, and has been applied on the simplified Chinese corpus PKU and MSR provided by SIGHAN Backoff2005. The F values obtained are 93.4% and 95. 5 respectively. According to the above method and domain adaptive word segmentation task, this paper proposes a word-based domain adaptive word segmentation method, because the word model is independent of each other. When the model is updated, the word model with strong migration performance is retained, and the word model with weak migration performance is updated and trained, which solves the difficulty and sharing of large-scale segmentation data. The source domain and target domain data mix need to be retrained and so on. When carries on the word segmentation to the target domain. The domain adaptation is realized by the adaptive ability of the model. The representation method of feature embedding can effectively solve the problem of feature sparsity. In this paper, feature embedding is used to represent input features. The experimental results show that. The word segmentation method proposed in this paper can obviously improve the adaptability of the field. Finally. The domain adaptive Chinese word segmentation system is designed and implemented. The system can use the existing basic model to segment the input sentence or text, and support the addition of relevant domain dictionaries. In addition, the basic model can be updated according to the training data of the domain to be partitioned, and the better segmentation results can be obtained.
【學位授予單位】：沈陽航空航天大學
【學位級別】：碩士
【學位授予年份】：2017
【分類號】：TP391.1

【參考文獻】

相關期刊論文 前10條

1 韓冰;劉一佳;車萬翔;劉挺;;基于感知器的中文分詞增量訓練方法研究[J];中文信息學報;2015年05期

2 韓冬煦;常寶寶;;中文分詞模型的領域適應性方法[J];計算機學報;2015年02期

3 周俊;鄭中華;張煒;;基于改進最大匹配算法的中文分詞粗分方法[J];計算機工程與應用;2014年02期

4 來斯惟;徐立恒;陳玉博;劉康;趙軍;;基于表示學習的中文分詞算法探索[J];中文信息學報;2013年05期

5 張梅山;鄧知龍;車萬翔;劉挺;;統(tǒng)計與詞典相結合的領域自適應中文分詞[J];中文信息學報;2012年02期

6 黃德根;焦世斗;周惠巍;;基于子詞的雙層CRFs中文分詞[J];計算機研究與發(fā)展;2010年05期

7 張桂平;劉東生;尹寶生;徐立軍;苗雪雷;;面向專利文獻的中文分詞技術的研究[J];中文信息學報;2010年03期

8 宋彥;蔡東風;張桂平;趙海;;一種基于字詞聯合解碼的中文分詞方法[J];軟件學報;2009年09期

9 黃昌寧;趙海;;中文分詞十年回顧[J];中文信息學報;2007年03期

10 秦穎;王小捷;張素香;;漢語分詞中組合歧義字段的研究[J];中文信息學報;2007年01期

，

本文編號：1466711