本文選題：大數(shù)據(jù) + 實(shí)時(shí)監(jiān)控��； 參考：《東華大學(xué)》2017年碩士論文

【摘要】：大數(shù)據(jù)時(shí)代,實(shí)時(shí)有效地收集海量訂單數(shù)據(jù),幫助企業(yè)智慧地從數(shù)據(jù)中獲取目標(biāo)信息,能夠更加有效的、有條理的制定相關(guān)業(yè)務(wù)的發(fā)展和改進(jìn)方向,企業(yè)可以通過(guò)對(duì)海量數(shù)據(jù)進(jìn)行二次開(kāi)發(fā)利用,進(jìn)一步地總結(jié)和處理數(shù)據(jù),最終制定出更加符合用戶(hù)需求、更加適用于市場(chǎng)的設(shè)計(jì)方案。Storm作為一套高效的、安全的、實(shí)時(shí)的大數(shù)據(jù)處理引擎被本系統(tǒng)使用,No SQL數(shù)據(jù)庫(kù)Elasticsearch和Mongo DB則能滿(mǎn)足對(duì)海量數(shù)據(jù)的高效存儲(chǔ)與查詢(xún)。本論文將基于Storm、Kafka、No SQL數(shù)據(jù)庫(kù)Elasticsearch和Mongo DB設(shè)計(jì)和實(shí)現(xiàn)訂單大數(shù)據(jù)的實(shí)時(shí)監(jiān)控系統(tǒng)。系統(tǒng)將主要的數(shù)據(jù)保存在Elasticsearch中,將一些配置參數(shù)和時(shí)間戳數(shù)據(jù)保存在Mongo DB中;文件系統(tǒng)使用HDFS;采用Scala語(yǔ)言來(lái)編寫(xiě)代碼;使用分布式消息隊(duì)列Kafka來(lái)連接Storm中不同功能的拓?fù)?提高了系統(tǒng)的可靠性。在數(shù)據(jù)處理過(guò)程中,執(zhí)行的操作名稱(chēng)、時(shí)間節(jié)點(diǎn)和部分中間結(jié)果會(huì)被記錄到系統(tǒng)日志中,以便解決系統(tǒng)錯(cuò)誤和提升系統(tǒng)性能,平臺(tái)處理后的重要結(jié)果,可以通過(guò)Web頁(yè)面以多種圖和表的形式向用戶(hù)展示,該展示網(wǎng)站具備搜索引擎,能夠通過(guò)關(guān)鍵字和特定規(guī)則的語(yǔ)句搜索目標(biāo)數(shù)據(jù),還支持點(diǎn)擊圖標(biāo)和標(biāo)簽進(jìn)行快捷查找。整套系統(tǒng)部署在分布式集群中,具有高實(shí)時(shí)性、高效率、高容錯(cuò)性、可擴(kuò)展等特點(diǎn),結(jié)果數(shù)據(jù)展示網(wǎng)站功能強(qiáng)大,界面清晰簡(jiǎn)潔,用戶(hù)體驗(yàn)很好,平臺(tái)可以實(shí)時(shí)地監(jiān)控海量數(shù)據(jù)信息的變化,從結(jié)構(gòu)混合、復(fù)雜的、規(guī)模龐大的數(shù)據(jù)中,通過(guò)智能化的方法,挖掘出有價(jià)值的信息,從而創(chuàng)造出一定的經(jīng)濟(jì)和社會(huì)價(jià)值。
[Abstract]:Big data era, real-time and effective collection of massive order data, to help enterprises intelligently obtain target information from the data, can be more effective, orderly development and improvement of related business direction,The enterprise can further sum up and process the data through the secondary development and utilization of the massive data, and finally work out a design scheme that is more in line with the needs of the user and more suitable for the market. Storm is a set of high efficiency and security.Using Elasticsearch and Mongo DB, the real-time big data processing engine can satisfy the high efficiency storage and query of massive data.This thesis will design and implement the real time monitoring system of order big data based on Elasticsearch and Mongo DB, which is based on Elasticsearch and Mongo DB.The system stores main data in Elasticsearch, saves some configuration parameters and timestamp data in Mongo DB, file system uses HDFS, uses Scala language to write code, uses distributed message queue Kafka to connect the topologies of different functions in Storm.The reliability of the system is improved.During data processing, the name of the operation performed, the time node, and some intermediate results are recorded in the system log to resolve system errors and improve system performance.Web pages can be displayed to users in the form of a variety of graphs and tables, the display site has a search engine, can search through keywords and specific rules of statements to search for target data, but also supports clicking on icons and labels for quick search.The whole system is deployed in the distributed cluster, with the characteristics of high real-time, high efficiency, high fault tolerance, extensibility, etc. The result data display website is powerful, the interface is clear and concise, and the user experience is very good.The platform can monitor the change of massive data information in real time. From the data of mixed structure, complex and large scale, through intelligent method, the platform can mine valuable information and create certain economic and social value.
【學(xué)位授予單位】：東華大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2017
【分類(lèi)號(hào)】：TP277

【參考文獻(xiàn)】

相關(guān)期刊論文 前1條

1 金澈清,錢(qián)衛(wèi)寧,周傲英;流數(shù)據(jù)分析與管理綜述[J];軟件學(xué)報(bào);2004年08期

，

本文編號(hào)：1731020