基于YARN和哈希技術的大數據K近鄰研究
發(fā)布時間：2018-12-16 21:15
【摘要】：大數據是近幾年機器學習領域最熱門的研究方向之一,大數據給傳統(tǒng)的機器學習帶來了巨大挑戰(zhàn)。K-近鄰是一種著名的分類算法。由于它簡單且易于實現,所以被廣泛應用于許多領域,如人臉識別、基因分類、決策支持等。然而,在大數據環(huán)境中,K-近鄰算法的效率變得非常低,甚至不可行。針對這一問題,基于Yarn和哈希技術,本文提出了兩種解決方案:一種用Mapreduce和SimHash在云計算平臺上實現針對大數據集的K-近鄰分類;另一種用Spark和SimHash在云計算平臺上實現針對大數據集的K-近鄰分類。兩種解決方案的基本思路是類似的,包括三步:(1)對大數據集做哈希變換,將其變換到海明空間;(2)在海明空間中,基于云計算Yarn平臺用大數據計算框架Mapreduce和Spark尋找與測試樣例x在同一個桶中的訓練樣例;(3)在同一個桶中再尋找測試樣例x的K個精確近鄰,并用這K個精確近鄰對x進行分類。實驗結果顯示,在分類能力保持的前提下,本文提出的解決方案是可行的,而且可以大幅度地提高K-近鄰算法的效率。
[Abstract]:Big data is one of the most popular research fields in the field of machine learning in recent years. Big data brings great challenges to the traditional machine learning. K- nearest neighbor is a famous classification algorithm. Because it is simple and easy to implement, it is widely used in many fields, such as face recognition, gene classification, decision support and so on. However, in big data environment, the efficiency of K-nearest neighbor algorithm becomes very low, even infeasible. Aiming at this problem, based on Yarn and hash technology, this paper proposes two solutions: one is to use Mapreduce and SimHash to realize K-nearest neighbor classification for big data set on cloud computing platform; Another is to use Spark and SimHash to implement K-nearest neighbor classification for big data set on cloud computing platform. The basic ideas of the two solutions are similar, including three steps: (1) Hash transformation of big data set and transform it into Heming space; (2) in Haiming space, based on cloud computing Yarn platform, big data computing framework Mapreduce and Spark are used to find and test sample x training samples in the same bucket; (3) the K exact nearest neighbors of test sample x are found in the same bucket, and the K exact nearest neighbors are used to classify x. The experimental results show that the proposed scheme is feasible and can greatly improve the efficiency of the K-nearest neighbor algorithm on the premise of maintaining the classification ability.
【學位授予單位】：河北大學
【學位級別】：碩士
【學位授予年份】：2017
【分類號】：TP311.13;TP181
本文編號：2383061
[Abstract]:Big data is one of the most popular research fields in the field of machine learning in recent years. Big data brings great challenges to the traditional machine learning. K- nearest neighbor is a famous classification algorithm. Because it is simple and easy to implement, it is widely used in many fields, such as face recognition, gene classification, decision support and so on. However, in big data environment, the efficiency of K-nearest neighbor algorithm becomes very low, even infeasible. Aiming at this problem, based on Yarn and hash technology, this paper proposes two solutions: one is to use Mapreduce and SimHash to realize K-nearest neighbor classification for big data set on cloud computing platform; Another is to use Spark and SimHash to implement K-nearest neighbor classification for big data set on cloud computing platform. The basic ideas of the two solutions are similar, including three steps: (1) Hash transformation of big data set and transform it into Heming space; (2) in Haiming space, based on cloud computing Yarn platform, big data computing framework Mapreduce and Spark are used to find and test sample x training samples in the same bucket; (3) the K exact nearest neighbors of test sample x are found in the same bucket, and the K exact nearest neighbors are used to classify x. The experimental results show that the proposed scheme is feasible and can greatly improve the efficiency of the K-nearest neighbor algorithm on the premise of maintaining the classification ability.
【學位授予單位】：河北大學
【學位級別】：碩士
【學位授予年份】：2017
【分類號】：TP311.13;TP181
【參考文獻】
相關期刊論文 前7條
1 黃宜華;;大數據機器學習系統(tǒng)研究進展[J];大數據;2015年01期
2 李武軍;周志華;;大數據哈希學習:現狀與趨勢[J];科學通報;2015年Z1期
3 陳潔;陳冬杰;黃幫明;;基于HBASE的大數據壓縮算法的研究[J];電腦知識與技術;2014年13期
4 張長水;;機器學習面臨的挑戰(zhàn)[J];中國科學:信息科學;2013年12期
5 姚吉龍;張瀟磊;;基于Hadoop的性能優(yōu)化分析[J];科技創(chuàng)新導報;2013年25期
6 閆永剛;馬廷淮;王建;;KNN分類算法的MapReduce并行化實現[J];南京航空航天大學學報;2013年04期
7 李國杰;程學旗;;大數據研究:未來科技及經濟社會發(fā)展的重大戰(zhàn)略領域——大數據的研究現狀與科學思考[J];中國科學院院刊;2012年06期
，本文編號：2383061
本文鏈接：http://sikaile.net/kejilunwen/zidonghuakongzhilunwen/2383061.html
