【摘要】：頻繁模式挖掘是數(shù)據(jù)挖掘領(lǐng)域的重要算法。頻繁模式挖掘在事務(wù)數(shù)據(jù)庫、時(shí)間序列數(shù)據(jù)庫和許多其他類型數(shù)據(jù)庫的挖掘研究中都得到了廣泛的應(yīng)用。然而,傳統(tǒng)的Frequent-pattern Growth算法(簡稱Fp-Growth算法)在處理大規(guī)模數(shù)據(jù)時(shí),無論是存儲上還是計(jì)算上都會遇到瓶頸,這就需要對Fp-Growth算法進(jìn)行并行化處理�，F(xiàn)有的并行Fp-Growth算法已經(jīng)解決了如何劃分?jǐn)?shù)據(jù)庫事務(wù)集這一問題,并保證了劃分后的事務(wù)集彼此之間相互獨(dú)立,但是現(xiàn)有的并行Fp-Growth算法和對事務(wù)集進(jìn)行劃分的時(shí)候缺乏了對負(fù)載均衡的考慮。因此,實(shí)現(xiàn)負(fù)載均衡的并行Fp-Growth算法是本文的主要問題。 Hadoop是Apache基金會下的一個(gè)開源的分布式并行編程框架,允許計(jì)算機(jī)集群通過使用簡單的編程模型分布式的處理大型數(shù)據(jù)集。Hadoop解決了并行計(jì)算存在的工作調(diào)度、分布式存儲、容錯(cuò)處理、網(wǎng)絡(luò)通訊等問題,這就使得開發(fā)者只需要關(guān)注算法本身,而系統(tǒng)本身的調(diào)度等問題都交由Hadoop處理�；谏鲜鲈�,所以本文使用Hadoop框架來實(shí)現(xiàn)并行化的Fp-Growth算法。 本文主要完成了以下兩個(gè)工作,一個(gè)是對現(xiàn)有的并行Fp-Growth算法提出了改進(jìn),另一個(gè)是將本文提出的并行算法應(yīng)用于頻繁用戶訪問序列的挖掘上。首先,本文在國內(nèi)外的并行Fp-Growth算法的研究基礎(chǔ)之上,利用估算每一個(gè)頻繁項(xiàng)的負(fù)載的方法,對現(xiàn)有的并行Fp-Growth算法的分組策略進(jìn)行了改進(jìn)。實(shí)驗(yàn)證明本文提出的改進(jìn)的并行Fp-Growth算法優(yōu)于現(xiàn)有的并行Fp-Growth算法,本文提出的算法具有更好的負(fù)載均衡能力和執(zhí)行效率。其次,由于Web服務(wù)器日志上存儲了海量的用戶訪問信息,因此,可以從海量的數(shù)據(jù)中發(fā)現(xiàn)那些隱藏起來的、有價(jià)值的用戶行為信息。所以,本文將提出的算法應(yīng)用于Web日志挖掘這一領(lǐng)域,用來挖掘頻繁的用戶訪問序列�；谶@一應(yīng)用方向所得到的結(jié)果可以對日志的來源網(wǎng)站提供指導(dǎo)和參考意見,具有實(shí)際的應(yīng)用價(jià)值和商業(yè)價(jià)值。
[Abstract]:Frequent pattern mining is an important algorithm in the field of data mining. Frequent pattern mining is widely used in the research of transaction database, time series database and many other kinds of database. However, the traditional Frequent-pattern Growth algorithm (Fp-Growth algorithm for short) will meet the bottleneck in both storage and computation when dealing with large-scale data, which requires parallelization of Fp-Growth algorithm. The existing parallel Fp-Growth algorithms have solved the problem of how to partition database transaction sets, and ensured that the partitioned transaction sets are independent of each other. However, the existing parallel Fp-Growth algorithms and transaction set partitioning lack of load balancing considerations. Therefore, the parallel Fp-Growth algorithm for load balancing is the main problem in this paper. Hadoop is an open source distributed parallel programming framework under the Apache Foundation, which allows computer clusters to deal with large data sets distributed by using simple programming models. Hadoop solves the problem of scheduling and distributed storage in parallel computing. Fault-tolerant processing, network communication and other problems, which make developers only need to pay attention to the algorithm itself, while the system itself scheduling problems are handled by Hadoop. For the above reasons, this paper uses Hadoop framework to implement parallel Fp-Growth algorithm. The main work of this paper is as follows: one is to improve the existing parallel Fp-Growth algorithm, the other is to apply the parallel algorithm to mining frequent user access sequences. Firstly, based on the research of the parallel Fp-Growth algorithm at home and abroad, this paper improves the grouping strategy of the existing parallel Fp-Growth algorithm by using the method of estimating the load of each frequent item. Experiments show that the improved parallel Fp-Growth algorithm is superior to the existing parallel Fp-Growth algorithm, and the proposed algorithm has better load balancing ability and execution efficiency. Secondly, because a large amount of user access information is stored in the Web server log, the hidden and valuable user behavior information can be found from the massive data. Therefore, the proposed algorithm is applied to the field of Web log mining, which is used to mine frequent user access sequences. Based on this application direction, the results can provide guidance and reference for the source websites of the log, and have practical application value and commercial value.
【學(xué)位授予單位】：山東大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2013
【分類號】：TP338.6

【參考文獻(xiàn)】

相關(guān)期刊論文 前10條

1 柴黃琪;蘇成;;基于HDFS的安全機(jī)制設(shè)計(jì)[J];計(jì)算機(jī)安全;2010年12期

2 劉永增;張曉景;李先毅;;基于Hadoop/Hive的web日志分析系統(tǒng)的設(shè)計(jì)[J];廣西大學(xué)學(xué)報(bào)(自然科學(xué)版);2011年S1期

3 黎宏劍;劉恒;黃廣文;卜立;;基于Hadoop的海量電信數(shù)據(jù)云計(jì)算平臺研究[J];電信科學(xué);2012年08期

4 陳文波;張秀娟;李林;唐鈞;;基于Hadoop的分布式日志分析系統(tǒng)[J];廣西大學(xué)學(xué)報(bào)(自然科學(xué)版);2011年S1期

5 黃濱;徐勇;呂巖;;基于Hadoop云存儲系統(tǒng)在設(shè)計(jì)院的應(yīng)用研究[J];信息安全與技術(shù);2012年09期

6 黃解軍,潘和平,萬幼川;數(shù)據(jù)挖掘技術(shù)的應(yīng)用研究[J];計(jì)算機(jī)工程與應(yīng)用;2003年02期

7 趙衛(wèi)中;馬慧芳;傅燕翔;史忠植;;基于云計(jì)算平臺Hadoop的并行k-means聚類算法設(shè)計(jì)研究[J];計(jì)算機(jī)科學(xué);2011年10期

8 李成華;張新訪;金海;向文;;MapReduce:新型的分布式并行計(jì)算編程模型[J];計(jì)算機(jī)工程與科學(xué);2011年03期

9 王振宇;郭力;;基于Hadoop的搜索引擎用戶行為分析[J];計(jì)算機(jī)工程與科學(xué);2011年04期

10 趙立江,何欽銘;一種個(gè)性化Web推薦系統(tǒng)的研究與實(shí)現(xiàn)[J];武漢理工大學(xué)學(xué)報(bào)(交通科學(xué)與工程版);2004年05期

相關(guān)碩士學(xué)位論文 前2條

1 楊雅雙;關(guān)聯(lián)規(guī)則的并行挖掘算法研究[D];西安科技大學(xué);2010年

2 楊銘馳;一種基于頻繁模式列表的關(guān)聯(lián)規(guī)則分類算法研究與實(shí)現(xiàn)[D];吉林大學(xué);2012年

，

本文編號：2390874