【摘要】：針對現(xiàn)有的Spark檢查點機制需要編程人員根據(jù)經(jīng)驗選擇檢查點,具有一定的風(fēng)險和隨機性,可能導(dǎo)致恢復(fù)開銷較大的問題,通過對RDD屬性的分析,提出了自動檢查點策略,包括權(quán)重生成(WG)算法和檢查點自動選擇(CAS)算法.首先,WG算法分析作業(yè)的DAG結(jié)構(gòu),獲取RDD的血統(tǒng)長度和操作復(fù)雜度等屬性,計算RDD權(quán)重;然后,CAS算法選擇權(quán)重大的RDD作為檢查點進(jìn)行異步備份,來實現(xiàn)數(shù)據(jù)的快速恢復(fù).結(jié)果表明:在使用CAS算法時,不同數(shù)據(jù)集執(zhí)行時間和檢查點容量大小都有所增加,其中Wiki-Talk由于其計算量較大,增幅明顯;使用CAS算法設(shè)置檢查點后,在單點失效恢復(fù)的情況下,數(shù)據(jù)集的恢復(fù)時間較短.因此,自動檢查點策略在略微增加執(zhí)行時間開銷的基礎(chǔ)上,能夠有效地降低作業(yè)的恢復(fù)開銷.
[Abstract]:In view of the fact that the existing Spark checkpoint mechanism needs the programmer to select the checkpoint according to the experience, it has certain risks and randomness, which may lead to the problem of large recovery overhead. Through the analysis of the RDD attribute, the automatic checkpoint strategy is put forward. Including weight generation (WG) algorithm and checkpoint automatic selection (CAS) algorithm. First, the WG algorithm analyzes the DAG structure of the job, acquires the properties of the RDD, such as the length of the lineage and the complexity of the operation, and calculates the RDD weight. Then, the CAS algorithm chooses RDD as a checkpoint to perform asynchronous backup to realize the fast data recovery. The results show that when CAS algorithm is used, the execution time and checkpoint capacity of different data sets are increased, among which, Wiki-Talk has a significant increase due to its large amount of calculation. After the checkpoint is set up by CAS algorithm, the recovery time of data set is shorter than that of single point failure recovery. Therefore, automatic checkpoint strategy can effectively reduce the cost of job recovery on the basis of a slight increase in execution time.
【作者單位】： 新疆大學(xué)信息科學(xué)與工程學(xué)院;新疆大學(xué)軟件學(xué)院;
【基金】：國家自然科學(xué)基金資助項目(61462079,61262088,61562086,61363083,61562078) 新疆維吾爾自治區(qū)高�？蒲杏媱澷Y助項目(XJEDU2016S106)
【分類號】：TP311.13

【相似文獻(xiàn)】

相關(guān)期刊論文 前2條

1 何鑫星;譚理;;數(shù)字地形圖質(zhì)量自動檢查方法探討及軟件開發(fā)[J];測繪;2013年04期

2 ;[J];;年期

，

本文編號：2417843