高并發(fā)下的全國零售戶信息采集和應用
發(fā)布時間：2018-08-01 11:32
【摘要】：截止目前企業(yè)面向的零售戶量已經(jīng)突破900萬,隨著信息化和數(shù)據(jù)化的加深,采集零售戶數(shù)據(jù)和信息也是勢在必行的。面對900萬的用戶量,隨之引起問題的是服務器的訪問量急劇上升,數(shù)據(jù)量呈爆炸式增長。以企業(yè)目前的服務器等硬件設備已經(jīng)無法滿足高并發(fā)、大數(shù)據(jù)的需求。如果只是增加多臺服務器組成服務器集群,提高硬件、軟件的性能,來解決大量用戶并發(fā)請求和大數(shù)據(jù)處理的問題,這顯然并不是很好的辦法。本文通過分析和對比傳統(tǒng)的數(shù)據(jù)采集模式和企業(yè)業(yè)務類型,設計了一種基于移動端的數(shù)據(jù)采集模式。并詳細的分析了該數(shù)據(jù)采集模型,為了滿足企業(yè)的要求和用戶的需求,針對新型采集模式面對的高并發(fā)、大數(shù)據(jù)處理的問題,進一步分析和研究了負載均衡調(diào)度策略,并根據(jù)基于高并發(fā)的特征和調(diào)度算法的特點,在多時間片輪詢調(diào)度機制的基礎上提出了基于任務請求預測的動態(tài)調(diào)整負載均衡算法。并通過分析多維Markov鏈和排隊論對算法進行進一步的調(diào)整和改進,經(jīng)過實驗驗證,該算法較好地提高了整體系統(tǒng)的性能和負載。具體研究內(nèi)容如下:1)本文通過對比傳統(tǒng)數(shù)據(jù)采集方法,根據(jù)目前企業(yè)的業(yè)務類型和數(shù)據(jù)請求特征,設計了一種基于分布式策略的新型數(shù)據(jù)采集模型,該模型將APP和微信企業(yè)號作為數(shù)據(jù)采集的入口,并且組建服務器集群對高并發(fā)業(yè)務進行調(diào)度處理。使用基于Hadoop平臺的策略創(chuàng)建了數(shù)據(jù)中心,并對數(shù)據(jù)進行了分析和處理。其中對全國零售戶請求的業(yè)務類型和提交的數(shù)據(jù)性質,對該模型進行了初步優(yōu)化處理。并通過分析模型系統(tǒng)中的層次結構和技術要點,提出了在高并發(fā)環(huán)境下,如何高效地調(diào)度任務請求和調(diào)整服務器節(jié)點的負載的問題,并在后文進行了分析和解決。2)深入研究了基于CPU和MEM的調(diào)度算法,根據(jù)實際任務請求的特征,改進并提出了基于預測機制的負載均衡調(diào)度算法。各個服務器節(jié)點會收集其它節(jié)點的負載情況,并預測網(wǎng)絡請求的業(yè)務類型和到達率,動態(tài)的調(diào)整請求的分發(fā),減少請求的等待時間,并縮短服務器的閑置時間,達到資源的有效利用,最終使系統(tǒng)的整體負載達到均衡狀態(tài)。通過實驗驗證,該算法在縮短響應時間方面具有良好的性能,并且比基于CPU和MEM的調(diào)度算法性能更好。3)針對基于預測機制的負載均衡算法本身存在的不足,利用排隊論的知識,對網(wǎng)絡服務器方面的負載進行了優(yōu)化,合理的安排任務請求的等待、處理和掛起。通過分析多維Markov的機制,對后續(xù)網(wǎng)絡請求的特征和聯(lián)系進行預測。通過分析多時間片輪詢策略,提出了基于預測的Markov排隊模型。經(jīng)過實驗的驗證和分析,該模型較好的協(xié)調(diào)各服務器節(jié)點的負載狀況,合理的分發(fā)后續(xù)到達的網(wǎng)絡請求,減小響應時間。4)根據(jù)數(shù)據(jù)采集模型和負載均衡技術,建立了一套集移動終端數(shù)據(jù)采集平臺、業(yè)務處理平臺、數(shù)據(jù)處理平臺的三位一體的數(shù)據(jù)采集系統(tǒng)。并將負載均衡技術應用到數(shù)據(jù)處理中,加快了服務器處理數(shù)據(jù)的時間,并展示了數(shù)據(jù)采集系統(tǒng)設計實現(xiàn)的成果。
[Abstract]:Up to now, the number of retail customers facing the enterprise has exceeded 9 million. With the deepening of information and data, it is imperative to collect data and information of retail customers. Facing 9 million users, the problem is that the amount of access to the server is rising rapidly and the amount of data is exploding. It is obviously not a good way to solve the problem of a large number of concurrent requests and large data processing, which is obviously not a very good solution. This article analyzes and compares the traditional data collection mode and enterprise business by analyzing and comparing the problems of large number of concurrent requests and large data processing. Type, a data acquisition model based on mobile terminal is designed, and the data acquisition model is analyzed in detail. In order to meet the requirements of the enterprise and the needs of the users, the load balancing scheduling strategy is further analyzed and studied for the problems of high concurrency and large data processing in the new collection mode. On the basis of the multi time chip polling scheduling mechanism, the dynamic adjustment load balancing algorithm based on the task request prediction is proposed. The algorithm is further adjusted and improved by analyzing the multidimensional Markov chain and queuing theory. The algorithm improves the performance and negative effect of the whole system. The specific research contents are as follows: 1) in this paper, a new data acquisition model based on the distributed strategy is designed by comparing the traditional data acquisition methods, according to the business types and data request features of the current enterprise. This model takes APP and WeChat enterprise number as the entrance of data collection, and sets up a server cluster for high concurrency services. The data center is created using the strategy based on the Hadoop platform, and the data are analyzed and processed. The model is optimized for the type of business and the nature of the data submitted by the national retail customers, and the high level structure and technical points in the model system are put forward. In the concurrent environment, how to efficiently schedule task requests and adjust the load of server nodes, and analyze and solve.2 in the later text, the scheduling algorithm based on CPU and MEM is deeply studied. According to the characteristics of the actual task request, the load balancing scheduling algorithm based on the prediction machine is improved and proposed. Collect the load of other nodes, predict the type of service and the rate of arrival of the network request, dynamically adjust the request distribution, reduce the waiting time of the request, shorten the idle time of the server, achieve the effective utilization of the resource, and finally make the whole load of the system reach the balance state. The inter aspect has good performance and is better than the scheduling algorithm based on CPU and MEM). In view of the shortcomings of the load balancing algorithm based on the prediction mechanism, using the knowledge of queuing theory, it optimizes the load of the network server and arranges the waiting, processing and hanging up. By analyzing multidimensional Ma. The mechanism of rkov predicts the characteristics and connections of the subsequent network requests. Through the analysis of the multi time slice polling strategy, a prediction based Markov queuing model is proposed. Through the experimental verification and analysis, the model is better to coordinate the load status of each server node, to distribute the following network requests reasonably, and to reduce the response time.4). According to the data collection model and the load balancing technology, a set of data collection system is set up, which sets the data collection platform of the mobile terminal, the business processing platform and the data processing platform. The load balancing technology is applied to the data processing, and the time for the server to deal with the data is speeded up, and the design of the data acquisition system is demonstrated. The present results.
【學位授予單位】：浙江理工大學
【學位級別】：碩士
【學位授予年份】：2017
【分類號】：F724.2
本文編號：2157441
[Abstract]:Up to now, the number of retail customers facing the enterprise has exceeded 9 million. With the deepening of information and data, it is imperative to collect data and information of retail customers. Facing 9 million users, the problem is that the amount of access to the server is rising rapidly and the amount of data is exploding. It is obviously not a good way to solve the problem of a large number of concurrent requests and large data processing, which is obviously not a very good solution. This article analyzes and compares the traditional data collection mode and enterprise business by analyzing and comparing the problems of large number of concurrent requests and large data processing. Type, a data acquisition model based on mobile terminal is designed, and the data acquisition model is analyzed in detail. In order to meet the requirements of the enterprise and the needs of the users, the load balancing scheduling strategy is further analyzed and studied for the problems of high concurrency and large data processing in the new collection mode. On the basis of the multi time chip polling scheduling mechanism, the dynamic adjustment load balancing algorithm based on the task request prediction is proposed. The algorithm is further adjusted and improved by analyzing the multidimensional Markov chain and queuing theory. The algorithm improves the performance and negative effect of the whole system. The specific research contents are as follows: 1) in this paper, a new data acquisition model based on the distributed strategy is designed by comparing the traditional data acquisition methods, according to the business types and data request features of the current enterprise. This model takes APP and WeChat enterprise number as the entrance of data collection, and sets up a server cluster for high concurrency services. The data center is created using the strategy based on the Hadoop platform, and the data are analyzed and processed. The model is optimized for the type of business and the nature of the data submitted by the national retail customers, and the high level structure and technical points in the model system are put forward. In the concurrent environment, how to efficiently schedule task requests and adjust the load of server nodes, and analyze and solve.2 in the later text, the scheduling algorithm based on CPU and MEM is deeply studied. According to the characteristics of the actual task request, the load balancing scheduling algorithm based on the prediction machine is improved and proposed. Collect the load of other nodes, predict the type of service and the rate of arrival of the network request, dynamically adjust the request distribution, reduce the waiting time of the request, shorten the idle time of the server, achieve the effective utilization of the resource, and finally make the whole load of the system reach the balance state. The inter aspect has good performance and is better than the scheduling algorithm based on CPU and MEM). In view of the shortcomings of the load balancing algorithm based on the prediction mechanism, using the knowledge of queuing theory, it optimizes the load of the network server and arranges the waiting, processing and hanging up. By analyzing multidimensional Ma. The mechanism of rkov predicts the characteristics and connections of the subsequent network requests. Through the analysis of the multi time slice polling strategy, a prediction based Markov queuing model is proposed. Through the experimental verification and analysis, the model is better to coordinate the load status of each server node, to distribute the following network requests reasonably, and to reduce the response time.4). According to the data collection model and the load balancing technology, a set of data collection system is set up, which sets the data collection platform of the mobile terminal, the business processing platform and the data processing platform. The load balancing technology is applied to the data processing, and the time for the server to deal with the data is speeded up, and the design of the data acquisition system is demonstrated. The present results.
【學位授予單位】：浙江理工大學
【學位級別】：碩士
【學位授予年份】：2017
【分類號】：F724.2
【參考文獻】
相關期刊論文 前10條
1 王小戲;吳剛;王灝;;高并發(fā)高可用零售O2O交易系統(tǒng)的架構設計與業(yè)務實現(xiàn)[J];計算機與現(xiàn)代化;2016年04期
2 林偉;丁志剛;;基于Agent的微信平臺自適應負載均衡算法[J];實驗技術與管理;2015年12期
3 喜藝;;Hadoop大數(shù)據(jù)平臺與傳統(tǒng)數(shù)據(jù)倉庫的協(xié)作探究[J];通訊世界;2015年17期
4 本刊編輯部;石菲;邢帆;孫杰賢;姜紅德;;“兩會”關鍵熱點:信息化新階段[J];中國信息化;2015年03期
5 王輝;李晉光;;異構網(wǎng)絡負載均衡算法[J];工業(yè)儀表與自動化裝置;2014年05期
6 蘇金樹;郭文忠;余朝龍;陳國龍;;負載均衡感知的無線傳感器網(wǎng)絡容錯分簇算法[J];計算機學報;2014年02期
7 黃素萍;葛萌;;Hadoop平臺在大數(shù)據(jù)處理中的應用研究[J];現(xiàn)代計算機(專業(yè)版);2013年29期
8 亓開元;韓燕波;趙卓峰;房俊;;支持高并發(fā)數(shù)據(jù)流處理的MapReduce中間結果緩存[J];計算機研究與發(fā)展;2013年01期
9 尚明華;秦磊磊;王風云;劉淑云;張曉艷;;基于Android智能手機的小麥生產(chǎn)風險信息采集系統(tǒng)[J];農(nóng)業(yè)工程學報;2011年05期
10 張濤;李建;康永佳;;多任務高并發(fā)數(shù)據(jù)處理平臺的技術研究[J];網(wǎng)絡安全技術與應用;2010年03期
相關博士學位論文 前2條
1 劉曉茜;云計算數(shù)據(jù)中心結構及其調(diào)度機制研究[D];中國科學技術大學;2011年
2 王永炎;實時事務并發(fā)控制算法優(yōu)化[D];中國科學院研究生院（軟件研究所）;2004年
相關碩士學位論文 前10條
1 鄧玉林;基于hadoop大數(shù)據(jù)框架的個性化推薦系統(tǒng)研究與實現(xiàn)[D];電子科技大學;2016年
2 張慧芳;基于動態(tài)反饋的加權最小連接數(shù)服務器負載均衡算法研究[D];華東理工大學;2013年
3 王延武;基于LVS的集群動態(tài)負載均衡算法研究[D];北京化工大學;2012年
4 馮秀玲;云計算環(huán)境下的負載均衡算法的研究與設計[D];北京郵電大學;2012年
5 胡俊;構建分布式系統(tǒng)的關鍵技術研究與實現(xiàn)[D];南京郵電大學;2012年
6 施楊斌;云計算環(huán)境下一種基于虛擬機動態(tài)遷移的負載均衡算法[D];復旦大學;2011年
7 楊錦;異構分布式系統(tǒng)中的負載均衡調(diào)度算法研究[D];湖南大學;2011年
8 萬勇;基于LVS的負載均衡策略算法的研究與改進[D];西南交通大學;2010年
9 劉漢玲;基于Web Service的分布式系統(tǒng)中動態(tài)負載均衡策略的研究與實現(xiàn)[D];哈爾濱工程大學;2010年
10 陳亮;集群負載均衡關鍵技術研究[D];中南大學;2009年
，本文編號：2157441
本文鏈接：http://sikaile.net/jingjilunwen/guojimaoyilunwen/2157441.html
最近更新
教材專著
