本文選題：工業(yè)數(shù)據(jù)壓縮　切入點：磁盤調(diào)度　出處：《南京信息工程大學》2014年碩士論文　論文類型：學位論文

【摘要】：隨著網(wǎng)格計算的高速發(fā)展,大規(guī)模實時工業(yè)數(shù)據(jù)分析在工業(yè)生產(chǎn)過程中起到了舉足輕重的作用,通過實時監(jiān)控分析工業(yè)數(shù)據(jù),能夠極大的指導生產(chǎn),提高生產(chǎn)力。然而在實時工業(yè)數(shù)據(jù)分析領域主要還存在幾大問題：第一,實時數(shù)據(jù)生成的速度非�？�,占用大量的系統(tǒng)存儲空間,如何在不丟失數(shù)據(jù)特征的情況下盡量減小存儲壓力是一個急需解決的問題。第二,在大規(guī)模的實時數(shù)據(jù)的傳輸及處理過程中,會在采集器與中心服務器間形成千上萬的并發(fā)數(shù)據(jù)連接,此時服務器的穩(wěn)定性、高效性至關重要。第三,傳統(tǒng)的TCP協(xié)議參數(shù)已經(jīng)不能適應當前的高速網(wǎng)絡,如何優(yōu)化協(xié)議使得網(wǎng)絡傳輸?shù)耐掏铝扛蟆Ｔ诒疚闹形覀兎謩e從數(shù)據(jù)壓縮、網(wǎng)格服務器性能以及網(wǎng)絡傳輸性能三個方面進行闡述我們網(wǎng)格計算相關的優(yōu)化成果。本文主要完成了以下工作： (1)工業(yè)數(shù)據(jù)實時性要求較高,會隨著時間形成一條曲線。工業(yè)產(chǎn)生的實時數(shù)據(jù)量非常大,不可以通過緩存大量數(shù)據(jù)后再分析整個數(shù)據(jù)的走向特性選擇對應的壓縮方案,整個壓縮算法的處理過程必須具有高壓縮比和低資源消耗。本文提出了針對工業(yè)實時數(shù)據(jù)壓縮的曲線面積映射壓縮法。根據(jù)測試結(jié)果,算法壓縮比是傳統(tǒng)SDT壓縮法的2.16倍。 (2)在數(shù)據(jù)網(wǎng)格環(huán)境中,服務器的性能一直是整體網(wǎng)格性能的關鍵要素之一。在本文中我們分析了網(wǎng)格服務器常見的性能瓶頸并在我們的網(wǎng)格環(huán)境中引入了用戶空間I/O調(diào)度、零拷貝、事件驅(qū)動架構(gòu)技術來提高網(wǎng)格服務器的性能。在大量小文件讀取的情況下引入用戶空間I/O調(diào)度能夠節(jié)省近50%的磁盤I/O時間。通過零拷貝,網(wǎng)格服務器可以減少CPU在大量內(nèi)核與用戶空間切換之間引起的消耗,節(jié)省63%無用的上下文切換。事件驅(qū)動架構(gòu)可以減少30%的CPU利用率并且達到線程驅(qū)動的最佳吞吐量。通過結(jié)合上面提到的這三種優(yōu)化方法應用到我們的網(wǎng)格服務器中,新的解決方案可以在只消耗傳統(tǒng)解決方案所占用CPU使用率的70%的情況下,系統(tǒng)飽和吞吐量比傳統(tǒng)方案提高30%。 (3)網(wǎng)絡流量測試表明,當前網(wǎng)絡95%的數(shù)據(jù)流為TCP流,其它的為UDP或其它形式的數(shù)據(jù)流,因此TCP協(xié)議傳輸性能成為了制約整個網(wǎng)絡數(shù)據(jù)傳輸性能的關鍵。通過分析TCP協(xié)議及其多個變種,縱向比較它們的擁塞控制算法,著重討論了影響TCP協(xié)議性能的網(wǎng)絡因素以及在不同網(wǎng)絡環(huán)境下TCP協(xié)議的網(wǎng)絡性能表現(xiàn)。實現(xiàn)了當前網(wǎng)絡傳輸常見場景下廣泛采用的相關TCP優(yōu)化方案,并提出根據(jù)神經(jīng)網(wǎng)絡來預測丟包在保持鏈路公平的前提下來提高TCP吞吐量的方案,通過網(wǎng)絡模擬實驗顯示,預測的一級命中率最高可以達到74%。
[Abstract]:With the rapid development of grid computing, large-scale real-time industrial data analysis plays an important role in the industrial production process, through real-time monitoring and analysis of industrial data, can greatly guide production, However, there are still several problems in the field of real-time industrial data analysis. First, the speed of real-time data generation is very fast, and it takes up a lot of system storage space. How to minimize the storage pressure without losing data features is an urgent problem. Second, in the process of large-scale real-time data transmission and processing, There will be thousands of concurrent data connections between the collector and the central server, so the stability and efficiency of the server is crucial. Third, the traditional TCP protocol parameters can no longer adapt to the current high-speed network. How to optimize the protocol to increase the throughput of network transport. Three aspects of grid server performance and network transmission performance are described in this paper. The main work of this paper is as follows:. (1) Industrial data requires high real-time performance and will form a curve with time. The amount of real time data produced by industry is very large, so it is not possible to choose the corresponding compression scheme by caching a large amount of data and then analyzing the trend characteristics of the whole data. The processing process of the whole compression algorithm must have high compression ratio and low resource consumption. In this paper, a curve area mapping compression method for industrial real-time data compression is proposed. According to the test results, the compression ratio of the algorithm is 2.16 times that of the traditional SDT compression method. In the data grid environment, the performance of the server is always one of the key elements of the overall grid performance. In this paper, we analyze the common performance bottlenecks of the grid server and introduce the user-space I / O scheduling into our grid environment. Zero copy, event driven architecture technology to improve the performance of grid servers. In the case of a large number of small files read, the introduction of user space I / O scheduling can save nearly 50% disk I / O time. Grid servers can reduce the cost of CPU between a large number of kernel and user space switching, Save 63% useless context switching. Event-driven architecture can reduce the CPU utilization of 30% and achieve the best thread-driven throughput. The new solution can increase the system saturation throughput by 30% over the traditional solution when only 70% of the CPU usage is consumed by the traditional solution. Network traffic tests show that the current network 95% data flow is TCP flow, others are UDP or other forms of data flow, Therefore, the transmission performance of TCP protocol has become the key to restrict the data transmission performance of the whole network. By analyzing the TCP protocol and its variety, the congestion control algorithm of TCP protocol is compared longitudinally. The network factors that affect the performance of TCP protocol and the performance of TCP protocol in different network environments are discussed emphatically, and the related TCP optimization schemes widely used in current network transmission scenarios are realized. A scheme is proposed to improve the throughput of TCP based on neural network to predict packet loss under the premise of maintaining the fairness of the link. The network simulation results show that the highest hit rate of the first class can reach 74%.
【學位授予單位】：南京信息工程大學
【學位級別】：碩士
【學位授予年份】：2014
【分類號】：TP333;TP393.05

【參考文獻】

相關期刊論文 前4條

1 劉紅霞;牛富麗;;實時數(shù)據(jù)庫數(shù)據(jù)壓縮算法探討與改進[J];化工自動化及儀表;2010年06期

2 錢笑宇,張彥武;工業(yè)實時數(shù)據(jù)庫的研究和設計[J];計算機工程;2005年01期

3 趙利強;于濤;王建林;;基于SQL數(shù)據(jù)庫的過程數(shù)據(jù)壓縮方法[J];計算機工程;2008年14期

4 熊永華;吳敏;賈維嘉;;基于延時預測的TCP實時視頻傳輸方法[J];中南大學學報(自然科學版);2010年04期

，

本文編號：1580931