【摘要】：風(fēng)力發(fā)電作為發(fā)展最快的新型清潔能源帶動(dòng)了風(fēng)電技術(shù)的發(fā)展和廣泛應(yīng)用并促使風(fēng)電機(jī)組的規(guī)模不斷擴(kuò)大,由此產(chǎn)生的數(shù)據(jù)規(guī)模也隨之?dāng)U大。在風(fēng)電機(jī)組處于工作狀態(tài)時(shí),其中的部件如齒輪箱、軸承等出現(xiàn)松動(dòng)、磨損、異常等都會(huì)產(chǎn)生大量的振動(dòng)數(shù)據(jù),難以滿足對(duì)海量數(shù)據(jù)整理、分析、存儲(chǔ)需求,而且由于風(fēng)電機(jī)的發(fā)電系統(tǒng)和監(jiān)控設(shè)備的多樣性,各類設(shè)備都產(chǎn)生不同的數(shù)據(jù)格式或數(shù)據(jù)類型,大都以數(shù)據(jù)流的形式輸出。因此諸如Hadoop等云計(jì)算平臺(tái)提供了對(duì)海量高維數(shù)據(jù)分析和處理的方式,為消耗大量資源的數(shù)據(jù)處理提供實(shí)時(shí)可靠相對(duì)廉價(jià)的計(jì)算資源。本文引入基于MapReduce的并行FFT算法和LZO壓縮相結(jié)合的技術(shù),對(duì)數(shù)據(jù)進(jìn)行處理以降低網(wǎng)絡(luò)傳輸量、減少存儲(chǔ)所用空間。快速傅里葉變換FFT實(shí)現(xiàn)了信號(hào)從復(fù)雜的時(shí)域轉(zhuǎn)換到具備顯著特征的頻域上,與離散傅里葉變換DFT相比大大減少了運(yùn)算量,在數(shù)字系統(tǒng)、計(jì)算機(jī)系統(tǒng)等信號(hào)處理方面FFT的廣泛應(yīng)用是一個(gè)重大進(jìn)展。FFT算法將數(shù)據(jù)以頻域形式展現(xiàn),能夠有效的分析數(shù)據(jù)的特征、設(shè)備狀態(tài)、進(jìn)行故障診斷等。本文研究了FFT算法的原理和特點(diǎn),采用在MapReduce的模型上實(shí)現(xiàn)了FFT的并行化。根據(jù)FFT的特點(diǎn)將算法的并行執(zhí)行分為數(shù)據(jù)補(bǔ)齊、變址運(yùn)算、蝶形運(yùn)算、格式化四個(gè)階段,為壓縮提供可靠的基礎(chǔ),加強(qiáng)數(shù)據(jù)存儲(chǔ)的效率。面對(duì)龐大數(shù)據(jù)量的存儲(chǔ),采用壓縮技術(shù)不僅節(jié)省空間,還可以降低數(shù)據(jù)文件傳輸過程中的I/O,本文在FFT每一階段的中間結(jié)果和最終結(jié)果中嵌入壓縮技術(shù),保證了數(shù)據(jù)壓縮的有效性。首先引入了Hadoop支持的Gzip、Bzip2、LZO三種壓縮格式,對(duì)三種壓縮格式的壓縮率和壓縮速度進(jìn)行測(cè)試比較得出了適用本文的壓縮格式,隨后著重研究了該壓縮方式的實(shí)現(xiàn)過程、分析壓縮性能,實(shí)現(xiàn)對(duì)振動(dòng)數(shù)據(jù)的壓縮存儲(chǔ)。
[Abstract]:Wind power, as the fastest developing new clean energy, promotes the development and wide application of wind power technology, and makes the scale of wind turbine expand constantly, and the resulting data scale also expands. When the wind turbine is in the working state, the components such as gearbox, bearing and so on will be loosened, worn, abnormal and so on will produce a large amount of vibration data, it is difficult to meet the demand of sorting, analyzing and storing the massive data. Because of the diversity of generating system and monitoring equipment of wind motor, all kinds of equipments produce different data format or data type, most of them are outputted in the form of data stream. Therefore cloud computing platforms such as Hadoop provide a way to analyze and process massive high-dimensional data and provide real-time reliable and relatively cheap computing resources for data processing which consumes a lot of resources. In this paper, the parallel FFT algorithm based on MapReduce and the technology of LZO compression are introduced to process the data in order to reduce the amount of network transmission and reduce the space used for storage. Fast Fourier transform (FFT) realizes signal conversion from complex time domain to frequency domain with obvious characteristics. Compared with discrete Fourier transform (DFT), it greatly reduces the computation. The wide application of FFT in signal processing, such as computer system, is a great progress. The algorithm presents the data in frequency domain, which can effectively analyze the characteristics of data, equipment status, fault diagnosis and so on. In this paper, the principle and characteristics of FFT algorithm are studied, and the parallelization of FFT based on MapReduce model is implemented. According to the characteristics of FFT, the parallel execution of the algorithm is divided into four stages: data completion, indexing, butterfly operation and formatting, which provide a reliable basis for compression and enhance the efficiency of data storage. In the face of the huge amount of data storage, the compression technology not only saves space, but also reduces the I / O ratio in the process of data file transfer. This paper embed compression technology in the intermediate and final results of each stage of FFT. The validity of data compression is ensured. Firstly, three compression schemes supported by Hadoop are introduced. The compression ratio and compression speed of the three compression formats are tested and compared, and the compression format suitable for this paper is obtained. Then, the realization process of the compression scheme is studied emphatically, and the compression performance is analyzed. The compression storage of vibration data is realized.
【學(xué)位授予單位】：華北電力大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2017
【分類號(hào)】：TM614

【參考文獻(xiàn)】

相關(guān)期刊論文 前10條

1 杜江;張錚;張杰鑫;邰銘;;MapReduce并行編程模型研究綜述[J];計(jì)算機(jī)科學(xué);2015年S1期

2 向麗輝;繆力;張大方;;壓縮對(duì)Hadoop性能影響研究[J];計(jì)算機(jī)工程與科學(xué);2015年02期

3 孫知信;黃涵霞;;基于云計(jì)算的數(shù)據(jù)存儲(chǔ)技術(shù)研究[J];南京郵電大學(xué)學(xué)報(bào)(自然科學(xué)版);2014年04期

4 羅鵬;龔勛;;HDFS數(shù)據(jù)存放策略的研究與改進(jìn)[J];計(jì)算機(jī)工程與設(shè)計(jì);2014年04期

5 李偉衛(wèi);趙航;張陽;王勇;;基于MapReduce的海量數(shù)據(jù)挖掘技術(shù)研究[J];計(jì)算機(jī)工程與應(yīng)用;2013年20期

6 翟巖龍;羅壯;楊凱;徐晟晨;;基于Hadoop的高性能海量數(shù)據(jù)處理平臺(tái)研究[J];計(jì)算機(jī)科學(xué);2013年03期

7 郝樹魁;;Hadoop HDFS和MapReduce架構(gòu)淺析[J];郵電設(shè)計(jì)技術(shù);2012年07期

8 周超;朱q，

本文編號(hào)：2168401