發(fā)布時(shí)間：2018-12-25 21:02

【摘要】：信息化技術(shù)在各行各業(yè)的普及,促使大規(guī)模數(shù)據(jù)產(chǎn)生于不同領(lǐng)域,給大數(shù)據(jù)處理帶來(lái)了全新的技術(shù)挑戰(zhàn)。高頻證券交易數(shù)據(jù)是典型的“流式大數(shù)據(jù)”,具有數(shù)據(jù)規(guī)模大、結(jié)構(gòu)復(fù)雜、流動(dòng)速度快等特點(diǎn)。如何利用有限的系統(tǒng)資源,構(gòu)建穩(wěn)定、可靠、高效的數(shù)據(jù)處理框架,在高頻推送的流式數(shù)據(jù)周期內(nèi)及時(shí)完成數(shù)據(jù)響應(yīng),是證券數(shù)據(jù)價(jià)值挖掘場(chǎng)景亟待解決的問(wèn)題。本文在分析研究大數(shù)據(jù)流式處理模型的基礎(chǔ)上,結(jié)合多種大數(shù)據(jù)處理技術(shù)構(gòu)建了一個(gè)面向高頻證券大數(shù)據(jù)流式處理框架,對(duì)其中涉及關(guān)鍵技術(shù)進(jìn)行研究和改進(jìn),并應(yīng)用于證券數(shù)據(jù)實(shí)時(shí)分析場(chǎng)景,實(shí)現(xiàn)了高效的數(shù)據(jù)流處理、管理與查詢(xún)。全文以構(gòu)建契合高頻證券大數(shù)據(jù)特征的流式數(shù)據(jù)處理框架為主線,并深入研究該框架中涉及的關(guān)鍵技術(shù),論文主要完成工作如下:(1)分析設(shè)計(jì)面向高頻證券大數(shù)據(jù)的流式處理框架。以Storm流式處理框架和Redis內(nèi)存數(shù)據(jù)庫(kù)為技術(shù)原型,將二者進(jìn)行結(jié)合并經(jīng)過(guò)改進(jìn),設(shè)計(jì)了面向高頻證券大數(shù)據(jù)的流式處理框架以及流式數(shù)據(jù)分層處理模型。(2)針對(duì)該框架中Storm組件的缺陷和不足,分別從物理、邏輯和應(yīng)用層面對(duì)Storm進(jìn)行優(yōu)化改進(jìn),以增強(qiáng)其面對(duì)高頻流式大數(shù)據(jù)的實(shí)時(shí)處理能力。(3)設(shè)計(jì)實(shí)現(xiàn)符合證券大數(shù)據(jù)高效存取的基于Redis的共享內(nèi)存中心。通過(guò)對(duì)Redis內(nèi)存數(shù)據(jù)庫(kù)的改進(jìn),既保留數(shù)據(jù)存儲(chǔ)的靈活性需求和可擴(kuò)展性?xún)?yōu)勢(shì),又考慮數(shù)據(jù)I/O的高效性,彌補(bǔ)了流式處理框架中Storm組件不能保存狀態(tài)數(shù)據(jù)的缺陷,為上層應(yīng)用的深度挖掘提供高效I/O保障。(4)本文設(shè)計(jì)的框架在高頻證券實(shí)時(shí)分析場(chǎng)景中的應(yīng)用。完成了面向高頻證券大數(shù)據(jù)的流式處理框架的應(yīng)用,為后續(xù)證券交易策略開(kāi)發(fā)和實(shí)現(xiàn)提供框架支撐。
[Abstract]:The popularization of information technology in various industries makes large-scale data come into being in different fields, which brings new technical challenges to big data. High-frequency securities trading data is a typical "flow big data" with the characteristics of large scale, complex structure and fast flow rate. How to make use of the limited system resources to construct a stable, reliable and efficient data processing framework, and to complete the data response timely in the high-frequency push flow data cycle, is an urgent problem to be solved in the securities data value mining scene. On the basis of analyzing and studying the large data flow processing model, this paper constructs a high frequency securities large data flow processing framework combined with various big data processing techniques, and researches and improves the key technologies involved in it. It is applied to real-time analysis of securities data to realize efficient data stream processing, management and query. The main line of this paper is to construct a flow data processing framework that fits the characteristics of high frequency securities big data, and to study the key technologies involved in the framework. The main work of this paper is as follows: (1) the flow processing framework for high frequency securities big data is analyzed and designed. Using Storm streaming processing framework and Redis memory database as the technical prototype, the two technologies are combined and improved. A streaming processing framework for high-frequency securities big data and a hierarchical model for streaming data processing are designed. (2) aiming at the defects and shortcomings of Storm components in this framework, the physical, logical and application layers are optimized and improved from the physical, logical and application layers to the Storm, respectively. In order to enhance its real-time processing ability to face high-frequency flow big data. (3) Design and implement the shared memory center based on Redis which conforms to the efficient access of securities big data. Through the improvement of Redis memory database, it not only preserves the flexibility and expansibility of data storage, but also considers the high efficiency of data I / O, which makes up for the defect that Storm component can not save state data in streaming processing framework. It provides an efficient I / O guarantee for the depth mining of the upper application. (4) the application of the framework designed in this paper in the real-time analysis scenario of high-frequency securities. The application of flow processing framework for high frequency securities big data is completed, which provides framework support for the development and implementation of subsequent securities trading strategies.
【學(xué)位授予單位】：西北大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2017
【分類(lèi)號(hào)】：F830.91;TP311.13

【參考文獻(xiàn)】

相關(guān)期刊論文 前7條

1 張琛;;證券投資基本分析及量化交易指標(biāo)的應(yīng)用研究[J];經(jīng)營(yíng)管理者;2015年09期

2 徐宗本;馮芷艷;郭迅華;曾大軍;陳國(guó)青;;大數(shù)據(jù)驅(qū)動(dòng)的管理與決策前沿課題[J];管理世界;2014年11期

3 程學(xué)旗;靳小龍;王元卓;郭嘉豐;張鐵贏;李國(guó)杰;;大數(shù)據(jù)系統(tǒng)和分析技術(shù)綜述[J];軟件學(xué)報(bào);2014年09期

4 孫大為;張廣艷;鄭緯民;;大數(shù)據(jù)流式計(jì)算:關(guān)鍵技術(shù)及系統(tǒng)實(shí)例[J];軟件學(xué)報(bào);2014年04期

5 Yaxiong Zhao;Jie Wu;Cong Liu;;Dache: A Data Aware Caching for Big-Data Applications Using the MapReduce Framework[J];Tsinghua Science and Technology;2014年01期

6 申德榮;于戈;王習(xí)特;聶鐵錚;寇月;;支持大數(shù)據(jù)管理的NoSQL系統(tǒng)研究綜述[J];軟件學(xué)報(bào);2013年08期

7 鄧建高;潘江波;楊出云;;基于數(shù)據(jù)倉(cāng)庫(kù)的證券交易決策支持系統(tǒng)設(shè)計(jì)與開(kāi)發(fā)[J];計(jì)算機(jī)工程與設(shè)計(jì);2008年03期

相關(guān)博士學(xué)位論文 前1條

1 歐陽(yáng)琳;分布式數(shù)據(jù)流處理系統(tǒng)動(dòng)態(tài)負(fù)載管理研究[D];武漢理工大學(xué);2009年

相關(guān)碩士學(xué)位論文 前2條

1 唐黎;面向金融大數(shù)據(jù)的高效數(shù)據(jù)處理機(jī)制的研究與設(shè)計(jì)[D];北京郵電大學(xué);2015年

2 顧昕;分布式流式計(jì)算框架關(guān)鍵技術(shù)的研究與實(shí)現(xiàn)[D];北京郵電大學(xué);2012年

，

本文編號(hào)：2391643