本文關(guān)鍵詞： 實時數(shù)據(jù)注入 事務(wù) 分布式數(shù)據(jù)庫 CLAIMS　出處：《華東師范大學(xué)》2017年碩士論文　論文類型：學(xué)位論文

【摘要】：隨著互聯(lián)網(wǎng)的發(fā)展,面對海量數(shù)據(jù)的存儲和管理問題,金融等行業(yè)開始采用分布式數(shù)據(jù)庫系統(tǒng)以提升系統(tǒng)的數(shù)據(jù)處理能力和可擴展性。為減少因數(shù)據(jù)注入延時和分析延時帶來的數(shù)據(jù)價值下降,金融等行業(yè)要求其系統(tǒng)支持實時的數(shù)據(jù)注入和查詢;同時也需要事務(wù)性的注入和查詢以保證金融數(shù)據(jù)的正確性和一致性。而傳統(tǒng)分布式環(huán)境下的事務(wù)處理大多基于鎖和兩階段提交,無法實現(xiàn)高通量實時數(shù)據(jù)注入,另一方面NoSQL分布式系統(tǒng)不支持事務(wù)型數(shù)據(jù)注入。CLAIMS系統(tǒng)是開源的分布式內(nèi)存OLAP系統(tǒng),支持實時數(shù)據(jù)查詢,但需要功能擴展以實現(xiàn)事務(wù)性實時數(shù)據(jù)注入。本文針對CLAIMS系統(tǒng),設(shè)計基于元數(shù)據(jù)的事務(wù)并發(fā)控制機制,進而提出分布式事務(wù)型實時數(shù)據(jù)注入框架,并通過大量實驗證明其滿足金融等領(lǐng)域的需求。本文的主要貢獻如下:1.針對CLAIMS系統(tǒng)設(shè)計基于元數(shù)據(jù)的事務(wù)并發(fā)控制機制,通過邏輯與物理相分離來達到控制元數(shù)據(jù)即控制事務(wù)數(shù)據(jù)的目的,原子性并發(fā)控制元數(shù)據(jù)以隔離寫事務(wù),利用事務(wù)快照以實現(xiàn)讀寫分離,支持強一致性。2.在實現(xiàn)分布式事務(wù)管理器的基礎(chǔ)上,本文設(shè)計事務(wù)型分布式實時數(shù)據(jù)注入引擎,廣泛使用無鎖結(jié)構(gòu)提供高吞吐量和低延時的實時數(shù)據(jù)注入,通過事務(wù)保證注入過程的ACID特性,且具有較高的可擴展性。3.在分布式OLAP系統(tǒng)上實現(xiàn)事務(wù)型實時數(shù)據(jù)注入框架,并通過一系列功能測試,與VoltDB進行性能對比測試,證明本文設(shè)計的框架在滿足事務(wù)特性的同時,能提供較高通量低時延的實時數(shù)據(jù)注入服務(wù)。本文提出的基于元數(shù)據(jù)的事務(wù)并發(fā)控制方式具有一定的創(chuàng)新性和學(xué)術(shù)價值,設(shè)計實現(xiàn)的實時數(shù)據(jù)注入基本滿足工業(yè)生產(chǎn)環(huán)境的需求,具有一定的實踐價值。本文的工作對我國金融行業(yè)的分布式數(shù)據(jù)實時注入具有借鑒意義。
[Abstract]:With the development of the Internet, facing the problem of massive data storage and management, In order to reduce the data value caused by data injection delay and analysis delay, finance and other industries began to adopt distributed database system to improve the data processing ability and extensibility of the system. Finance and other industries require their systems to support real-time data injection and query; At the same time, transactional injection and query are also needed to ensure the correctness and consistency of financial data. In the traditional distributed environment, most transactions are based on locks and two-phase commit, so high-throughput real-time data injection can not be realized. On the other hand, NoSQL distributed system does not support transactional data injection. CLAIMS is an open source distributed memory OLAP system, which supports real-time data query, but it needs function expansion to realize transactional real-time data injection. The transaction concurrency control mechanism based on metadata is designed, and a distributed transactional real-time data injection framework is proposed. The main contributions of this paper are as follows: 1. The transaction concurrency control mechanism based on metadata is designed for CLAIMS system. The purpose of controlling metadata, i.e. controlling transaction data, is achieved by separating logic from physics. Atomic concurrent control metadata is used to isolate write transactions, and transaction snapshots are used to realize the separation of reading and writing. Based on the implementation of distributed transaction manager, this paper designs a transactional distributed real-time data injection engine, which provides high throughput and low latency real-time data injection. The transactional real-time data injection framework is implemented on the distributed OLAP system, and a series of functional tests are carried out to compare the performance of the framework with that of VoltDB. It is proved that the proposed framework can provide real-time data injection services with high throughput and low delay while satisfying the transaction characteristics. The proposed transaction concurrency control method based on metadata has some innovative and academic value. The design and implementation of real-time data injection basically meet the needs of the industrial production environment and have certain practical value. The work of this paper has reference significance for the distributed data real-time injection in the financial industry of our country.
【學(xué)位授予單位】：華東師范大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2017
【分類號】：TP311.13

【參考文獻】

相關(guān)期刊論文 前2條

1 鄒先霞;賈維嘉;潘久輝;;基于數(shù)據(jù)庫日志的變化數(shù)據(jù)捕獲研究[J];小型微型計算機系統(tǒng);2012年03期

2 陸劍峰,張浩;數(shù)據(jù)倉庫數(shù)據(jù)更新的研究及基于Oracle數(shù)據(jù)庫的開發(fā)與應(yīng)用[J];計算機工程與應(yīng)用;2004年26期

相關(guān)碩士學(xué)位論文 前1條

1 張磊;Claims系統(tǒng)中的大表連接算法研究與實現(xiàn)[D];華東師范大學(xué);2015年

，

本文編號：1551219