循環(huán)注意力模型的訓練優(yōu)化策略研究
發(fā)布時間：2018-11-09 13:15
【摘要】：近年來,深度學習在計算機視覺,機器翻譯,語音識別等領域取得了極大的成功,在多個應用領域上取得了當前的最好成績。但是這些模型所取得的高精確度主要來源于在訓練以及檢測時投入了高額計算成本。傳統(tǒng)的深度學習的一個主要運算瓶頸在于需要對整幅圖像處理,而人類視覺只需要將視覺焦點集中在當前感興趣的區(qū)域上,這一特點能夠很有效的減少人類視覺系統(tǒng)的“帶寬”。在視覺領域中,盡管研究人員提出了如減少滑動窗口的方法以提高計算效率,但是深度模型的計算成本依然跟輸入圖像的大小成正比。為了解決該問題,本文模擬人類視覺系統(tǒng)特點引入注意力機制。目前的注意力機制主要分為Soft Attention以及Hard Attention。Soft Attention基于顯著圖的可微模型,而Hard Attention采用離散的注意力位置產(chǎn)生凝視(Glimpse)區(qū)域來產(chǎn)生注意力特征。本文從Hard Attention出發(fā),基于循環(huán)注意力模型(RAM)提出了兩種優(yōu)化策略——OV-RAM以及EM算法,并且在弱標記的數(shù)據(jù)集Translated MNIST以及Cluttered MNIST上進行了測試。循環(huán)注意力模型基于RNN,能夠每次凝視不同的感知區(qū)域從而更新自身隱含狀態(tài),通過這些累積信息來做決策。由于每次只處理感興趣的小部分區(qū)域,所以運算效率比傳統(tǒng)深度學習網(wǎng)絡更高。由于循環(huán)注意力采用了離散的、不可導的注意力位置,并且運用了強化學習來求解注意力位置選擇策略,這使得循環(huán)注意力模型訓練速率較慢。本文借鑒前人的模型,將Soft Attention與Hard Attention結(jié)合,給循環(huán)注意力模型加入Overview層以提供上下文信息,得到OV-RAM模型。同時,本文分析了循環(huán)注意力模型結(jié)構(gòu)中的問題,從有監(jiān)督學習的角度重新推導了目標函數(shù),將兩個耦合部分拆分,從而引入EM算法對進行訓練。最后,我們對一些失敗樣例進行了分析,并給出了一些解決措施。本文使用Translated MNIST以及Cluttered MNIST數(shù)據(jù)集進行訓練及測試。實驗結(jié)果證實,本文提出的OV-RAM以及EM算法能夠有效的提升循環(huán)注意力模型的訓練速率。該方法只需要更短的迭代次數(shù)就可達到同樣的收斂精度,證明了本文提出的兩種優(yōu)化策略的有效性。
[Abstract]:In recent years, deep learning has achieved great success in computer vision, machine translation, speech recognition and so on. However, the high accuracy of these models is mainly due to the high cost of training and detection. One of the main operational bottlenecks of traditional in-depth learning is the need to process the entire image, while human vision only needs to focus on the current region of interest. This feature can effectively reduce the "bandwidth" of human visual systems. In the field of vision, although researchers have proposed methods such as reducing sliding windows to improve computational efficiency, the computational cost of the depth model is still proportional to the size of the input image. In order to solve this problem, the attention mechanism is introduced to simulate the characteristics of human visual system. At present, the attention mechanism is mainly divided into Soft Attention and Hard Attention.Soft Attention differentiable model based on salient graph, while Hard Attention uses discrete attention position to produce gaze (Glimpse) region to produce attention feature. In this paper, based on Hard Attention, two optimization strategies, OV-RAM and EM algorithm, are proposed based on the cyclic attention model (RAM) and tested on the weakly marked dataset Translated MNIST and Cluttered MNIST. The cyclic attention model is based on the fact that RNN, can gaze into different perceptual regions each time to update its own implicit state and make decisions through these cumulative information. Since only a small part of the region of interest is processed at a time, the computational efficiency is higher than that of the traditional deep learning network. Because cyclic attention adopts discrete and underivable attention positions and reinforcement learning is used to solve the attention position selection strategy, the training rate of cyclic attention model is slow. Based on the previous models, this paper combines Soft Attention with Hard Attention, adds Overview layer to the cyclic attention model to provide context information, and obtains the OV-RAM model. At the same time, this paper analyzes the problems in the structure of the cyclic attention model, rededuces the objective function from the perspective of supervised learning, splits the two coupling parts, and then introduces the EM algorithm to train. Finally, we analyze some examples of failure and give some solutions. This paper uses Translated MNIST and Cluttered MNIST data sets for training and testing. Experimental results show that the proposed OV-RAM and EM algorithms can effectively improve the training rate of the cyclic attention model. The method can achieve the same convergence accuracy only with shorter iterations, which proves the effectiveness of the two optimization strategies proposed in this paper.
【學位授予單位】：哈爾濱工業(yè)大學
【學位級別】：碩士
【學位授予年份】：2017
【分類號】：TP391.41
本文編號：2320514
[Abstract]:In recent years, deep learning has achieved great success in computer vision, machine translation, speech recognition and so on. However, the high accuracy of these models is mainly due to the high cost of training and detection. One of the main operational bottlenecks of traditional in-depth learning is the need to process the entire image, while human vision only needs to focus on the current region of interest. This feature can effectively reduce the "bandwidth" of human visual systems. In the field of vision, although researchers have proposed methods such as reducing sliding windows to improve computational efficiency, the computational cost of the depth model is still proportional to the size of the input image. In order to solve this problem, the attention mechanism is introduced to simulate the characteristics of human visual system. At present, the attention mechanism is mainly divided into Soft Attention and Hard Attention.Soft Attention differentiable model based on salient graph, while Hard Attention uses discrete attention position to produce gaze (Glimpse) region to produce attention feature. In this paper, based on Hard Attention, two optimization strategies, OV-RAM and EM algorithm, are proposed based on the cyclic attention model (RAM) and tested on the weakly marked dataset Translated MNIST and Cluttered MNIST. The cyclic attention model is based on the fact that RNN, can gaze into different perceptual regions each time to update its own implicit state and make decisions through these cumulative information. Since only a small part of the region of interest is processed at a time, the computational efficiency is higher than that of the traditional deep learning network. Because cyclic attention adopts discrete and underivable attention positions and reinforcement learning is used to solve the attention position selection strategy, the training rate of cyclic attention model is slow. Based on the previous models, this paper combines Soft Attention with Hard Attention, adds Overview layer to the cyclic attention model to provide context information, and obtains the OV-RAM model. At the same time, this paper analyzes the problems in the structure of the cyclic attention model, rededuces the objective function from the perspective of supervised learning, splits the two coupling parts, and then introduces the EM algorithm to train. Finally, we analyze some examples of failure and give some solutions. This paper uses Translated MNIST and Cluttered MNIST data sets for training and testing. Experimental results show that the proposed OV-RAM and EM algorithms can effectively improve the training rate of the cyclic attention model. The method can achieve the same convergence accuracy only with shorter iterations, which proves the effectiveness of the two optimization strategies proposed in this paper.
【學位授予單位】：哈爾濱工業(yè)大學
【學位級別】：碩士
【學位授予年份】：2017
【分類號】：TP391.41
【相似文獻】
相關碩士學位論文 前1條
1 陳少鵬;循環(huán)注意力模型的訓練優(yōu)化策略研究[D];哈爾濱工業(yè)大學;2017年
，本文編號：2320514
本文鏈接：http://sikaile.net/kejilunwen/ruanjiangongchenglunwen/2320514.html
最近更新
教材專著
