基于統(tǒng)計和規(guī)則的中文人名識別研究與實現(xiàn)
發(fā)布時間：2018-08-17 10:38
【摘要】：中文分詞技術的研究是中文信息處理的一項基礎性課題,廣泛應用于搜索引擎、機器翻譯、信息抽取、文本聚類等領域。目前,影響分詞質(zhì)量的主要因素是歧義切分和對未登錄詞的識別,而人名在未登錄詞中又是數(shù)量最多、識別難度最大的一類,分詞系統(tǒng)中往往針對人名有專門的模塊進行識別。提高對人名識別的質(zhì)量,不僅能夠提高分詞的精度,而且對信息抽取和詞法分析有很大幫助。 本文針對現(xiàn)代漢語文本,主要研究人名的自動識別問題。在對大規(guī)模姓名樣本庫和語料庫進行統(tǒng)計的基礎上,對人名用字和人名邊界詞進行分析,總結人名用字和人名邊界詞出現(xiàn)規(guī)律,使用基于相對可信度的統(tǒng)計模型和針對系統(tǒng)自身特點設計的一系列規(guī)則來進行人名識別。具體地,本文的主要工作有三方面內(nèi)容：一是對人名識別所使用的資源作分析,對大規(guī)模人名庫(含480萬個人名)和語料庫(累計詞頻30億)進行統(tǒng)計,總結人名用字特點和規(guī)律,對人名的邊界信息作了詳細分析,根據(jù)人名邊界詞的詞性和所表達的意義對其進行了分級,作為人名外部屬性幫助人名識別,然后對本文所使用的百科語料庫與傳統(tǒng)語料庫進行了對比,指出其優(yōu)越性；在本文所使用的統(tǒng)計方法方面,使用基于相對可信度的統(tǒng)計模型對大規(guī)模語料庫進行了統(tǒng)計,同時對兩種特殊形式的人名建立了模型并作出統(tǒng)計,建立了人名各類用字的統(tǒng)計信息表；在規(guī)則方法的使用方面,本文設計了一系列的規(guī)則用于提取候選姓名和對人名識別結果進行校正。最后本文通過統(tǒng)計獲得系統(tǒng)使用的各個閾值和參數(shù),通過實驗對在研究過程中使用的方法做了對比,并驗證本文所使用的統(tǒng)計模型和規(guī)則的有效性。 對1998年1月份《人民日報》語料庫進行測試,實驗結果表明,本系統(tǒng)獲得了較高的準確率和召回率,人名識別獲得了良好的效果,提高了整個分詞系統(tǒng)的精度。
[Abstract]:The research of Chinese word segmentation is a basic subject of Chinese information processing, which is widely used in search engine, machine translation, information extraction, text clustering and so on. At present, the main factors that affect the quality of word segmentation are ambiguous segmentation and recognition of unrecorded words, but the number of unrecorded words is the largest and the recognition is the most difficult. In the word segmentation system, there is a special module for the recognition of people's names. Improving the quality of human name recognition can not only improve the accuracy of word segmentation, but also help information extraction and lexical analysis. This paper focuses on the automatic recognition of human names in modern Chinese texts. On the basis of the statistics of large scale name sample database and corpus, this paper analyzes the character of human name and the boundary word of person name, and sums up the rule of appearance of the word of name and boundary word of person name. Based on the statistical model of relative credibility and a series of rules designed according to the characteristics of the system, name recognition is carried out. Specifically, the main work of this paper has three aspects: the first is to analyze the resources used in the identification of people's names, and to make statistics on the large-scale names bank (including 4.8 million names) and the corpus (cumulative word frequency 3 billion). This paper summarizes the characteristics and rules of characters used in personal names, analyzes the boundary information of names in detail, classifies them according to their parts of speech and their meanings, and helps them to recognize their names as the external attributes of names. Then, the paper compares the encyclopedia corpus with the traditional corpus, points out its superiority, and uses the statistical model based on the relative credibility to calculate the large-scale corpus in the statistical methods used in this paper. At the same time, the model and statistics of two special forms of names are established, and the statistical information tables of all kinds of characters are established. In this paper, a series of rules are designed to extract candidate names and correct the recognition results. Finally, the threshold and parameters of the system are obtained by statistics, and the methods used in the research are compared through experiments, and the validity of the statistical model and rules used in this paper is verified. The People's Daily corpus in January 1998 was tested. The experimental results show that the system has a high accuracy and recall rate, and the recognition of human names has a good effect and improves the accuracy of the whole word segmentation system.
【學位授予單位】：西南交通大學
【學位級別】：碩士
【學位授予年份】：2013
【分類號】：TP391.1
[Abstract]:The research of Chinese word segmentation is a basic subject of Chinese information processing, which is widely used in search engine, machine translation, information extraction, text clustering and so on. At present, the main factors that affect the quality of word segmentation are ambiguous segmentation and recognition of unrecorded words, but the number of unrecorded words is the largest and the recognition is the most difficult. In the word segmentation system, there is a special module for the recognition of people's names. Improving the quality of human name recognition can not only improve the accuracy of word segmentation, but also help information extraction and lexical analysis. This paper focuses on the automatic recognition of human names in modern Chinese texts. On the basis of the statistics of large scale name sample database and corpus, this paper analyzes the character of human name and the boundary word of person name, and sums up the rule of appearance of the word of name and boundary word of person name. Based on the statistical model of relative credibility and a series of rules designed according to the characteristics of the system, name recognition is carried out. Specifically, the main work of this paper has three aspects: the first is to analyze the resources used in the identification of people's names, and to make statistics on the large-scale names bank (including 4.8 million names) and the corpus (cumulative word frequency 3 billion). This paper summarizes the characteristics and rules of characters used in personal names, analyzes the boundary information of names in detail, classifies them according to their parts of speech and their meanings, and helps them to recognize their names as the external attributes of names. Then, the paper compares the encyclopedia corpus with the traditional corpus, points out its superiority, and uses the statistical model based on the relative credibility to calculate the large-scale corpus in the statistical methods used in this paper. At the same time, the model and statistics of two special forms of names are established, and the statistical information tables of all kinds of characters are established. In this paper, a series of rules are designed to extract candidate names and correct the recognition results. Finally, the threshold and parameters of the system are obtained by statistics, and the methods used in the research are compared through experiments, and the validity of the statistical model and rules used in this paper is verified. The People's Daily corpus in January 1998 was tested. The experimental results show that the system has a high accuracy and recall rate, and the recognition of human names has a good effect and improves the accuracy of the whole word segmentation system.
【學位授予單位】：西南交通大學
【學位級別】：碩士
【學位授予年份】：2013
【分類號】：TP391.1
【參考文獻】
相關期刊論文 前10條
1 黃德根,馬玉霞,楊元生;基于互信息的中文姓名識別方法[J];大連理工大學學報;2004年05期
2 李建華,王曉龍;中文人名自動識別的一種有效方法[J];高技術通訊;2000年02期
3 毋琳;鄭逢斌;喬保軍;湯賽麗;;HENU漢語分詞系統(tǒng)中的中文人名識別算法[J];計算機工程與應用;2006年14期
4 賈品貴;楊一平;盧朋;;基于統(tǒng)計方法的中文姓名識別研究[J];計算機工程與應用;2006年31期
5 曹波;蘇一丹;鄧琦;;基于最大熵模型的中國人名自動識別[J];計算機工程與應用;2009年04期
6 張騰飛;王曉磊;王保云;;基于場景信息融合的中文姓名識別方法研究[J];計算機工程與應用;2009年34期
7 王源媛;何中市;;基于詞性探測的中文姓名識別算法[J];計算機科學;2005年04期
8 高紅;黃德根;楊元生;;一種與分詞一體化的中文人名識別方法[J];計算機工程;2006年19期
9 李麗雙;黃德根;毛婷婷;徐瀟瀟;;基于支持向量機的中國人名的自動識別[J];計算機工程;2006年19期
10 賈寧;張全;;基于最大熵模型的中文姓名識別[J];計算機工程;2007年09期
相關會議論文 前1條
1 季Y，
本文編號：2187344
本文鏈接：http://sikaile.net/kejilunwen/sousuoyinqinglunwen/2187344.html
最近更新
教材專著
