支持分庫分表和讀寫分離的ORM框架的設計與實現(xiàn)
發(fā)布時間：2018-08-13 11:18
【摘要】：隨著互聯(lián)網(wǎng)的飛速發(fā)展,越來越多的企業(yè)在百度搜索引擎上投放廣告,實現(xiàn)流量變現(xiàn)。所以廣告業(yè)務系統(tǒng)的數(shù)據(jù)呈現(xiàn)爆炸性的增長,單一的數(shù)據(jù)庫服務器已經(jīng)不足以存儲所有的數(shù)據(jù),所以按照一系列規(guī)則對數(shù)據(jù)庫做了分庫分表的處理,并提供主從庫來分離數(shù)據(jù)庫的讀寫操作,以提高讀操作的性能。而在廣告業(yè)務系統(tǒng)中需要一個統(tǒng)一的模塊去和所有的數(shù)據(jù)庫進行交互。雖然目前開源的對象映射(ORM)框架已經(jīng)很多,但是這些框架屬于通用型的,不能和業(yè)務及數(shù)據(jù)庫最好地配合。而系統(tǒng)目前使用的ORM框架性能較低,對緩存服務器內(nèi)存的使用比較浪費。為了更好地和數(shù)據(jù)庫、緩存等交互,并節(jié)省緩存服務器的內(nèi)存,提出支持數(shù)據(jù)庫分庫分表以及讀寫分離的ORM框架的需求。該框架使用JAVA語言開發(fā),并和MYSQL服務器和REDIS緩存服務器進行交互。其設計上分為六個模塊:配置管理、數(shù)據(jù)源管理、映射管理、緩存管理、數(shù)據(jù)存取和數(shù)據(jù)報告。其中配置管理完成了數(shù)據(jù)庫和緩存配置文件的定義和加載。數(shù)據(jù)源管理完成了如何將數(shù)據(jù)庫和緩存的配置解析并注冊為相應的數(shù)據(jù)源對象,并如何集中管理和查找數(shù)據(jù)源。映射管理完成了如何通過JAVA注解將JAVA的對象映射到數(shù)據(jù)庫表。緩存管理完成了緩存的結(jié)構(gòu)以及如何通過一致性哈希管理緩存服務器。數(shù)據(jù)存取完成了數(shù)據(jù)庫和緩存的增刪改查功能的實現(xiàn),并完成了數(shù)據(jù)庫讀寫分離的實現(xiàn)和如何通過分布式緩存解決數(shù)據(jù)庫的主從延遲。作為框架的核心,其性能會影響系統(tǒng)的處理能力,所以提升數(shù)據(jù)庫和緩存操作的性能成為關鍵。最后完成了數(shù)據(jù)報告模塊,通過輸出框架運行時的各項數(shù)據(jù)指標,協(xié)助分析框架的性能和緩存使用情況�？蚣苓�通過JAVASSIST字節(jié)碼框架和多線程大大提高了框架的性能。目前完成了框架的所有需求,并通過模擬線上請求進行了功能測試和性能測試。目前已經(jīng)上線并穩(wěn)定運行一段時間,在并發(fā)量較大的情況下也能大大地提升性能,而對緩存服務器內(nèi)存的使用也大大減少。
[Abstract]:With the rapid development of the Internet, more and more enterprises put advertisements on Baidu search engine to realize traffic realization. So the data of the advertising business system showed explosive growth, a single database server is no longer enough to store all the data, so according to a series of rules, the database is divided into databases and tables. And provides master-slave library to separate database read and write operations, to improve the performance of reading operations. In the advertising business system, a unified module is needed to interact with all databases. Although there are many open source object mapping (ORM) frameworks, these frameworks are generic and can not best match business and database. At present, the ORM framework used in the system has low performance and is a waste of cache server memory. In order to better interact with database, cache and save memory of cache server, the paper puts forward the requirement of supporting database sorting and reading and writing separate ORM framework. The framework is developed using the JAVA language and interacts with MYSQL servers and REDIS cache servers. The design is divided into six modules: configuration management, data source management, mapping management, cache management, data access and data reporting. Configuration management completes the definition and loading of database and cache configuration files. Data source management completes how to parse and register the configuration of database and cache as the corresponding data source object, and how to centrally manage and find the data source. Mapping management completes how to map JAVA objects to database tables through JAVA annotations. Cache management completes the structure of cache and how to manage cache server through consistency hash. Data access completes the functions of database and cache modification and deletion, and realizes the separation of database reading and writing and how to solve the master-slave delay of database by distributed cache. As the core of the framework, its performance will affect the processing ability of the system, so improving the performance of database and cache operation becomes the key. Finally, the data reporting module is completed, and the performance and cache usage of the framework are analyzed by outputting the data indexes of the framework. The framework also greatly improves the performance of the framework through JAVASSIST bytecode framework and multi-thread. At present, all the requirements of the framework have been completed, and functional and performance tests have been carried out by simulating online requests. At present, it has been running steadily for a period of time, and can greatly improve the performance under the condition of large amount of concurrency, and reduce the memory usage of the cache server greatly.
【學位授予單位】：哈爾濱工業(yè)大學
【學位級別】：碩士
【學位授予年份】：2016
【分類號】：TP311.13
本文編號：2180821
[Abstract]:With the rapid development of the Internet, more and more enterprises put advertisements on Baidu search engine to realize traffic realization. So the data of the advertising business system showed explosive growth, a single database server is no longer enough to store all the data, so according to a series of rules, the database is divided into databases and tables. And provides master-slave library to separate database read and write operations, to improve the performance of reading operations. In the advertising business system, a unified module is needed to interact with all databases. Although there are many open source object mapping (ORM) frameworks, these frameworks are generic and can not best match business and database. At present, the ORM framework used in the system has low performance and is a waste of cache server memory. In order to better interact with database, cache and save memory of cache server, the paper puts forward the requirement of supporting database sorting and reading and writing separate ORM framework. The framework is developed using the JAVA language and interacts with MYSQL servers and REDIS cache servers. The design is divided into six modules: configuration management, data source management, mapping management, cache management, data access and data reporting. Configuration management completes the definition and loading of database and cache configuration files. Data source management completes how to parse and register the configuration of database and cache as the corresponding data source object, and how to centrally manage and find the data source. Mapping management completes how to map JAVA objects to database tables through JAVA annotations. Cache management completes the structure of cache and how to manage cache server through consistency hash. Data access completes the functions of database and cache modification and deletion, and realizes the separation of database reading and writing and how to solve the master-slave delay of database by distributed cache. As the core of the framework, its performance will affect the processing ability of the system, so improving the performance of database and cache operation becomes the key. Finally, the data reporting module is completed, and the performance and cache usage of the framework are analyzed by outputting the data indexes of the framework. The framework also greatly improves the performance of the framework through JAVASSIST bytecode framework and multi-thread. At present, all the requirements of the framework have been completed, and functional and performance tests have been carried out by simulating online requests. At present, it has been running steadily for a period of time, and can greatly improve the performance under the condition of large amount of concurrency, and reduce the memory usage of the cache server greatly.
【學位授予單位】：哈爾濱工業(yè)大學
【學位級別】：碩士
【學位授予年份】：2016
【分類號】：TP311.13
【相似文獻】
相關期刊論文 前9條
1 ;得分表[J];個人電腦;2004年03期
2 何琳;何娟;閻素蘭;;基于自動標引的《中分表》改造及測評研究[J];中國圖書館學報;2012年06期
3 曹玉強;;《中分表》第二版使用過程中若干標引問題的探討[J];國家圖書館學刊;2009年01期
4 ;得分表[J];個人電腦;2003年02期
5 周兵;;談《中分表》Web版?zhèn)�人名稱主題詞的注釋[J];圖書館工作與研究;2010年08期
6 王正發(fā);;用Excel打造智能成績統(tǒng)分表[J];中小學電教;2006年10期
7 湯少清;檔案牽引知青情[J];湖南檔案;2001年04期
8 孫國新;鄉(xiāng)站應用計算機前景廣闊[J];農(nóng)村電氣化;1994年03期
9 ;[J];;年期
相關重要報紙文章 前1條
1 記者 楊澤元 通訊員 管曉梅;分表計費降低能耗[N];中國石油報;2006年
相關碩士學位論文 前1條
1 嚴成武;支持分庫分表和讀寫分離的ORM框架的設計與實現(xiàn)[D];哈爾濱工業(yè)大學;2016年
，本文編號：2180821
本文鏈接：http://sikaile.net/wenyilunwen/guanggaoshejilunwen/2180821.html
最近更新
教材專著
