基于內(nèi)存優(yōu)化配置的MapReduce性能調(diào)優(yōu)
發(fā)布時(shí)間：2019-01-23 09:10
【摘要】：MapReduce作業(yè)性能與內(nèi)存配置存在極大的相關(guān)性,針對(duì)準(zhǔn)確預(yù)測(cè)作業(yè)內(nèi)存困難問(wèn)題,根據(jù)Java虛擬機(jī)(JVM)的分代內(nèi)存管理特點(diǎn),提出了一種分代內(nèi)存預(yù)測(cè)方法.首先使用回歸模型對(duì)年輕代與垃圾回收平均時(shí)間的關(guān)系進(jìn)行建模,將尋找合理年輕代內(nèi)存大小的問(wèn)題轉(zhuǎn)換為一個(gè)受約束的非線性優(yōu)化問(wèn)題,并設(shè)計(jì)搜索算法來(lái)求解該優(yōu)化問(wèn)題.文中還建立MapReduce作業(yè)的Map任務(wù)和Reduce任務(wù)性能與內(nèi)存的關(guān)系模型,求解最佳性能的內(nèi)存需求,從而獲得Map任務(wù)和Reduce任務(wù)的年長(zhǎng)代內(nèi)存大小;使用聚類算法預(yù)測(cè)JVM晉升對(duì)象閾值,優(yōu)化JVM配置,減少了JVM的垃圾回收暫停時(shí)間.實(shí)驗(yàn)結(jié)果表明,文中提出的方法能準(zhǔn)確預(yù)測(cè)作業(yè)的內(nèi)存需求,顯著提升作業(yè)運(yùn)行性能.
[Abstract]:There is a great correlation between MapReduce job performance and memory configuration. Aiming at the difficulty of accurately predicting job memory, a generation memory prediction method is proposed according to the characteristics of generation memory management of Java Virtual Machine (JVM). First, the relationship between the young generation and the average garbage collection time is modeled by using the regression model. The problem of finding the reasonable memory size of the younger generation is transformed into a constrained nonlinear optimization problem, and a search algorithm is designed to solve the optimization problem. In this paper, the relationship model of Map task and Reduce task performance and memory of MapReduce job is established to solve the memory requirement of the best performance, so as to obtain the older generation memory size of Map task and Reduce task. The clustering algorithm is used to predict the threshold of JVM promotion object, optimize the JVM configuration, and reduce the garbage collection pause time of JVM. The experimental results show that the proposed method can accurately predict the memory requirements of jobs and significantly improve the performance of jobs.
【作者單位】： 四川大學(xué)網(wǎng)絡(luò)空間安全研究院;
【基金】：國(guó)家科技支撐計(jì)劃項(xiàng)目(2012BAH18B05) 國(guó)家自然科學(xué)基金資助項(xiàng)目(61272447)~~
【分類號(hào)】：TP311.13
本文編號(hào)：2413657
[Abstract]:There is a great correlation between MapReduce job performance and memory configuration. Aiming at the difficulty of accurately predicting job memory, a generation memory prediction method is proposed according to the characteristics of generation memory management of Java Virtual Machine (JVM). First, the relationship between the young generation and the average garbage collection time is modeled by using the regression model. The problem of finding the reasonable memory size of the younger generation is transformed into a constrained nonlinear optimization problem, and a search algorithm is designed to solve the optimization problem. In this paper, the relationship model of Map task and Reduce task performance and memory of MapReduce job is established to solve the memory requirement of the best performance, so as to obtain the older generation memory size of Map task and Reduce task. The clustering algorithm is used to predict the threshold of JVM promotion object, optimize the JVM configuration, and reduce the garbage collection pause time of JVM. The experimental results show that the proposed method can accurately predict the memory requirements of jobs and significantly improve the performance of jobs.
【作者單位】： 四川大學(xué)網(wǎng)絡(luò)空間安全研究院;
【基金】：國(guó)家科技支撐計(jì)劃項(xiàng)目(2012BAH18B05) 國(guó)家自然科學(xué)基金資助項(xiàng)目(61272447)~~
【分類號(hào)】：TP311.13
【相似文獻(xiàn)】
相關(guān)期刊論文 前8條
1 楊霖;DOS技巧集錦[J];電腦愛好者;1994年09期
2 李志峰;在DECpcXL486DX2服務(wù)器上安裝NOVELL3．11[J];中國(guó)金融電腦;1995年02期
3 張可心;WPS2000反片輸出[J];電腦愛好者;2001年02期
4 ;優(yōu)化漫談[J];電腦采購(gòu)周刊;2001年05期
5 朱運(yùn)喜;;談?wù)刉indows中的內(nèi)存管理[J];電腦采購(gòu)周刊;2001年21期
6 沈洪;Xteam Linux安裝詳解[J];電子科技;1999年18期
7 周鵬;讓W(xué)in98跑得更快[J];大眾用電;2000年01期
8 ;[J];;年期
，本文編號(hào)：2413657
本文鏈接：http://sikaile.net/kejilunwen/ruanjiangongchenglunwen/2413657.html
最近更新
教材專著
