【摘要】：針對現(xiàn)有的Spark檢查點(diǎn)機(jī)制需要編程人員根據(jù)經(jīng)驗(yàn)選擇檢查點(diǎn),具有一定的風(fēng)險(xiǎn)和隨機(jī)性,可能導(dǎo)致恢復(fù)開銷較大的問題,通過對RDD屬性的分析,提出了自動(dòng)檢查點(diǎn)策略,包括權(quán)重生成(WG)算法和檢查點(diǎn)自動(dòng)選擇(CAS)算法.首先,WG算法分析作業(yè)的DAG結(jié)構(gòu),獲取RDD的血統(tǒng)長度和操作復(fù)雜度等屬性,計(jì)算RDD權(quán)重;然后,CAS算法選擇權(quán)重大的RDD作為檢查點(diǎn)進(jìn)行異步備份,來實(shí)現(xiàn)數(shù)據(jù)的快速恢復(fù).結(jié)果表明:在使用CAS算法時(shí),不同數(shù)據(jù)集執(zhí)行時(shí)間和檢查點(diǎn)容量大小都有所增加,其中Wiki-Talk由于其計(jì)算量較大,增幅明顯;使用CAS算法設(shè)置檢查點(diǎn)后,在單點(diǎn)失效恢復(fù)的情況下,數(shù)據(jù)集的恢復(fù)時(shí)間較短.因此,自動(dòng)檢查點(diǎn)策略在略微增加執(zhí)行時(shí)間開銷的基礎(chǔ)上,能夠有效地降低作業(yè)的恢復(fù)開銷.
[Abstract]:In view of the fact that the existing Spark checkpoint mechanism needs the programmer to select the checkpoint according to the experience, it has certain risks and randomness, which may lead to the problem of large recovery overhead. Through the analysis of the RDD attribute, the automatic checkpoint strategy is put forward. Including weight generation (WG) algorithm and checkpoint automatic selection (CAS) algorithm. First, the WG algorithm analyzes the DAG structure of the job, acquires the properties of the RDD, such as the length of the lineage and the complexity of the operation, and calculates the RDD weight. Then, the CAS algorithm chooses RDD as a checkpoint to perform asynchronous backup to realize the fast data recovery. The results show that when CAS algorithm is used, the execution time and checkpoint capacity of different data sets are increased, among which, Wiki-Talk has a significant increase due to its large amount of calculation. After the checkpoint is set up by CAS algorithm, the recovery time of data set is shorter than that of single point failure recovery. Therefore, automatic checkpoint strategy can effectively reduce the cost of job recovery on the basis of a slight increase in execution time.
【作者單位】： 新疆大學(xué)信息科學(xué)與工程學(xué)院;新疆大學(xué)軟件學(xué)院;
【基金】：國家自然科學(xué)基金資助項(xiàng)目(61462079,61262088,61562086,61363083,61562078) 新疆維吾爾自治區(qū)高�？蒲杏�(jì)劃資助項(xiàng)目(XJEDU2016S106)
【分類號(hào)】：TP311.13

【相似文獻(xiàn)】

相關(guān)期刊論文 前2條

1 何鑫星;譚理;;數(shù)字地形圖質(zhì)量自動(dòng)檢查方法探討及軟件開發(fā)[J];測繪;2013年04期

2 ;[J];;年期

，

本文編號(hào)：2417843