手機閱讀平臺數(shù)據(jù)倉庫管理模塊的設(shè)計與實現(xiàn)
發(fā)布時間：2019-05-26 20:42
【摘要】：隨著信息技術(shù)的不斷發(fā)展,企業(yè)面臨的數(shù)據(jù)愈加豐富和繁雜,其計算機系統(tǒng)產(chǎn)生的數(shù)據(jù)也越來越多,數(shù)據(jù)倉庫技術(shù)的不斷成熟,為企業(yè)的數(shù)據(jù)管理提供了有效的解決方案。在數(shù)據(jù)倉庫的搭建、使用、建設(shè)中,隨著海量數(shù)據(jù)的收集、入庫、加工計算,數(shù)據(jù)資產(chǎn)的組織管理和日常訪問的復(fù)雜性,成為人們關(guān)注的焦點。對于使用者而言,他們面對多個異構(gòu)的數(shù)據(jù)來源,指標(biāo)和解釋有差異、統(tǒng)計口徑不一致,業(yè)務(wù)人員的理解與開發(fā)人員具體實現(xiàn)不同步;對于業(yè)務(wù)和技術(shù)人員來說,他們要應(yīng)對多重系統(tǒng),業(yè)務(wù)名詞的定義沒有同業(yè)務(wù)、系統(tǒng)的發(fā)展相適應(yīng),缺少標(biāo)準(zhǔn)的信息承載平臺。如何構(gòu)建健全的數(shù)據(jù)倉庫管理系統(tǒng),是企業(yè)面對復(fù)雜數(shù)據(jù)資產(chǎn),需要解決的重點問題。目前中國移動手機閱讀基地的Hadoop Hive數(shù)據(jù)倉庫平臺已經(jīng)穩(wěn)定運營將近一年,開發(fā)任務(wù)基本都只在倉庫的應(yīng)用層面上,滿足業(yè)務(wù)應(yīng)用需求,然而倉庫自身的管理、維護(hù)沒有形成完整的功能集,用戶只能依賴手工翻閱大量文本去獲取所需信息。為了方便后期對整個數(shù)據(jù)倉庫平臺的科學(xué)管理與維護(hù),本課題設(shè)計并實現(xiàn)了符合其自身特點的數(shù)據(jù)倉庫管理模塊。良好的倉庫管理模塊的設(shè)計不僅便于IT人員、技術(shù)人員及維護(hù)人員更好地管理和使用數(shù)據(jù)倉庫資源,而且可以在很大程度上幫助普通業(yè)務(wù)人員靈活利用倉庫所提供的海量數(shù)據(jù)。本課題設(shè)計并開發(fā)符合手機閱讀基地Hadoop Hive數(shù)據(jù)倉庫自身特點的倉庫管理系統(tǒng),提供元數(shù)據(jù)管理、任務(wù)調(diào)度監(jiān)控、數(shù)據(jù)血緣分析功能。元數(shù)據(jù)管理使得用戶可以高效的找出其所關(guān)心的數(shù)據(jù),同時也是任務(wù)調(diào)度監(jiān)控和數(shù)據(jù)血緣分析的基礎(chǔ)。任務(wù)調(diào)度監(jiān)控使得用戶可以實時獲取現(xiàn)網(wǎng)Hive運行狀況,了解其上、下游節(jié)點信息和運行狀況。數(shù)據(jù)血緣分析向用戶提供數(shù)據(jù)地圖,用戶可以方便了解數(shù)據(jù)的來源和去向,并為后續(xù)倉庫結(jié)構(gòu)優(yōu)化提供可靠的數(shù)據(jù)支撐。論文組織結(jié)構(gòu)如下:第一章為緒論,介紹了本課題的研究背景、研究內(nèi)容、研究意義,以及在手機閱讀項目中的具體應(yīng)用。第二章對本課題涉及的相關(guān)背景和技術(shù)進(jìn)行簡要介紹,包括手機閱讀BI(商業(yè)智能)平臺、數(shù)據(jù)倉庫、元數(shù)據(jù)、Oozie和數(shù)據(jù)倉庫應(yīng)用現(xiàn)狀。第三章對手機閱讀平臺數(shù)據(jù)倉庫管理模塊進(jìn)行需求分析與總體設(shè)計。第四章介紹了倉庫管理模塊的詳細(xì)設(shè)計和實現(xiàn)方案,包括各個關(guān)鍵模塊實現(xiàn)的方案。第五章為系統(tǒng)測試與分析,對各項功能進(jìn)行測試,并對系統(tǒng)應(yīng)用前后的效果進(jìn)行了對比分析。第六章針對本課題工作進(jìn)行總結(jié),對下一步的研究工作進(jìn)行展望。
[Abstract]:With the continuous development of information technology, enterprises are facing more and more abundant and complicated data, and more data are generated by its computer system. the continuous maturity of data warehouse technology provides an effective solution for enterprise data management. In the construction, use and construction of data warehouse, with the collection, storage, processing and calculation of massive data, the organization and management of data assets and the complexity of daily access, people have become the focus of attention. For users, they face multiple heterogeneous data sources, indicators and interpretations are different, statistical caliber is inconsistent, and the understanding of business personnel is out of sync with the specific implementation of developers. For business and technical personnel, they have to deal with multiple systems, the definition of business terms is not in line with the development of business, system development, lack of standard information bearing platform. How to construct a sound data warehouse management system is a key problem that enterprises need to solve in the face of complex data assets. At present, the Hadoop Hive data warehouse platform of China Mobile phone Reading Base has been stable for nearly a year, and the development tasks are basically only on the application level of the warehouse to meet the business application needs. However, the warehouse itself is managed. Maintenance does not form a complete feature set, users can only rely on manual reading a large number of text to obtain the required information. In order to facilitate the scientific management and maintenance of the whole data warehouse platform in the later stage, a data warehouse management module in accordance with its own characteristics is designed and implemented in this paper. The design of a good warehouse management module is not only convenient for IT personnel, technicians and maintenance personnel to better manage and use data warehouse resources, but also can help ordinary business personnel to make flexible use of the massive data provided by the warehouse to a great extent. This paper designs and develops a warehouse management system which accords with the characteristics of Hadoop Hive data warehouse in mobile phone reading base, which provides the functions of metadata management, task scheduling monitoring and data consanguinity analysis. Metadata management enables users to find out the data they care about efficiently, and it is also the basis of task scheduling monitoring and data consanguinity analysis. Task scheduling monitoring enables users to obtain the running status of Hive in real time, and to understand the information and running status of upstream and downstream nodes. Data consanguinity analysis provides users with data maps, which can easily understand the source and direction of data, and provide reliable data support for subsequent warehouse structure optimization. The organizational structure of the paper is as follows: the first chapter is the introduction, which introduces the research background, research content, research significance, and the specific application in mobile phone reading project. The second chapter briefly introduces the related background and technology of this topic, including mobile phone reading BI (Business Intelligence) platform, data warehouse, metadata, Oozie and data warehouse application status. The third chapter carries on the requirement analysis and the overall design to the mobile phone reading platform data warehouse management module. The fourth chapter introduces the detailed design and implementation of warehouse management module, including the implementation of each key module. The fifth chapter is the system test and analysis, carries on the test to each function, and has carried on the comparative analysis to the system before and after the application effect. The sixth chapter summarizes the work of this topic and looks forward to the next research work.
【學(xué)位授予單位】：北京郵電大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2016
【分類號】：TP311.52
本文編號：2485614
[Abstract]:With the continuous development of information technology, enterprises are facing more and more abundant and complicated data, and more data are generated by its computer system. the continuous maturity of data warehouse technology provides an effective solution for enterprise data management. In the construction, use and construction of data warehouse, with the collection, storage, processing and calculation of massive data, the organization and management of data assets and the complexity of daily access, people have become the focus of attention. For users, they face multiple heterogeneous data sources, indicators and interpretations are different, statistical caliber is inconsistent, and the understanding of business personnel is out of sync with the specific implementation of developers. For business and technical personnel, they have to deal with multiple systems, the definition of business terms is not in line with the development of business, system development, lack of standard information bearing platform. How to construct a sound data warehouse management system is a key problem that enterprises need to solve in the face of complex data assets. At present, the Hadoop Hive data warehouse platform of China Mobile phone Reading Base has been stable for nearly a year, and the development tasks are basically only on the application level of the warehouse to meet the business application needs. However, the warehouse itself is managed. Maintenance does not form a complete feature set, users can only rely on manual reading a large number of text to obtain the required information. In order to facilitate the scientific management and maintenance of the whole data warehouse platform in the later stage, a data warehouse management module in accordance with its own characteristics is designed and implemented in this paper. The design of a good warehouse management module is not only convenient for IT personnel, technicians and maintenance personnel to better manage and use data warehouse resources, but also can help ordinary business personnel to make flexible use of the massive data provided by the warehouse to a great extent. This paper designs and develops a warehouse management system which accords with the characteristics of Hadoop Hive data warehouse in mobile phone reading base, which provides the functions of metadata management, task scheduling monitoring and data consanguinity analysis. Metadata management enables users to find out the data they care about efficiently, and it is also the basis of task scheduling monitoring and data consanguinity analysis. Task scheduling monitoring enables users to obtain the running status of Hive in real time, and to understand the information and running status of upstream and downstream nodes. Data consanguinity analysis provides users with data maps, which can easily understand the source and direction of data, and provide reliable data support for subsequent warehouse structure optimization. The organizational structure of the paper is as follows: the first chapter is the introduction, which introduces the research background, research content, research significance, and the specific application in mobile phone reading project. The second chapter briefly introduces the related background and technology of this topic, including mobile phone reading BI (Business Intelligence) platform, data warehouse, metadata, Oozie and data warehouse application status. The third chapter carries on the requirement analysis and the overall design to the mobile phone reading platform data warehouse management module. The fourth chapter introduces the detailed design and implementation of warehouse management module, including the implementation of each key module. The fifth chapter is the system test and analysis, carries on the test to each function, and has carried on the comparative analysis to the system before and after the application effect. The sixth chapter summarizes the work of this topic and looks forward to the next research work.
【學(xué)位授予單位】：北京郵電大學(xué)
【學(xué)位級別】：碩士
【學(xué)位授予年份】：2016
【分類號】：TP311.52
【參考文獻(xiàn)】
相關(guān)期刊論文 前3條
1 張明治;;基于CWM規(guī)范設(shè)計的元數(shù)據(jù)管理系統(tǒng)[J];電腦知識與技術(shù);2014年02期
2 楊鴻賓;宋明;;元數(shù)據(jù)管理平臺總體架構(gòu)設(shè)計研究[J];計算機系統(tǒng)應(yīng)用;2007年11期
3 王磊;李一凡;趙懷慈;;銀聯(lián)數(shù)據(jù)倉庫系統(tǒng)中ETL的設(shè)計和實現(xiàn)[J];微電子學(xué)與計算機;2007年05期
相關(guān)博士學(xué)位論文 前1條
1 魏建生;高性能重復(fù)數(shù)據(jù)檢測與刪除技術(shù)研究[D];華中科技大學(xué);2012年
相關(guān)碩士學(xué)位論文 前4條
1 任桂禾;大數(shù)據(jù)處理支撐平臺調(diào)度子系統(tǒng)的設(shè)計與實現(xiàn)[D];北京郵電大學(xué);2015年
2 朱斌;基于Hadoop的日志統(tǒng)計分析系統(tǒng)的設(shè)計與實現(xiàn)[D];哈爾濱工業(yè)大學(xué);2013年
3 毛瑞雪;基于數(shù)據(jù)血緣的審計證據(jù)追蹤技術(shù)研究及應(yīng)用[D];哈爾濱工程大學(xué);2012年
4 賈文娟;基于hive分布式計算與數(shù)據(jù)挖掘的關(guān)聯(lián)性營銷的設(shè)計與實現(xiàn)[D];北京交通大學(xué);2011年
，本文編號：2485614
本文鏈接：http://sikaile.net/kejilunwen/ruanjiangongchenglunwen/2485614.html
最近更新
教材專著
