本文關(guān)鍵詞：激活函數(shù)導(dǎo)向的RNN算法優(yōu)化　出處：《浙江大學(xué)》2017年碩士論文　論文類型：學(xué)位論文

  更多相關(guān)文章： 循環(huán)神經(jīng)網(wǎng)絡(luò) 長(zhǎng)短時(shí)記憶單元 激活函數(shù) 非飽和區(qū) 線性擬合

【摘要】：循環(huán)神經(jīng)網(wǎng)絡(luò)(RecurrentNeuralNetworks,RNN)是人工神經(jīng)網(wǎng)絡(luò)的一個(gè)重要分支。它在隱含層引入了反饋機(jī)制,實(shí)現(xiàn)對(duì)序列數(shù)據(jù)的有效處理。循環(huán)神經(jīng)網(wǎng)絡(luò)具有存儲(chǔ)和處理上下文信息的強(qiáng)大能力,成為語(yǔ)音識(shí)別、自然語(yǔ)言處理、計(jì)算機(jī)視覺(jué)等領(lǐng)域的研究熱點(diǎn)之一。一方面,循環(huán)神經(jīng)網(wǎng)絡(luò)普遍采用S型函數(shù)作為激活函數(shù),而S型函數(shù)的飽和區(qū)限制了RNN訓(xùn)練收斂速度,因此對(duì)激活函數(shù)的優(yōu)化研究成為研究熱點(diǎn)。另一方面,循環(huán)神經(jīng)網(wǎng)絡(luò)主要采用軟件實(shí)現(xiàn)的方式,算法的硬件加速研究具有重要意義。本文針對(duì)上述問(wèn)題和研究背景,在前人的研究基礎(chǔ)上做了如下工作:循環(huán)神經(jīng)網(wǎng)絡(luò)的理論總結(jié)研究。長(zhǎng)短時(shí)記憶單元((Long Short-Term Memory,LSTM)特有的門(mén)結(jié)構(gòu)解決了傳統(tǒng)循環(huán)神經(jīng)網(wǎng)絡(luò)時(shí)間維度的梯度消失問(wèn)題,成為RNN結(jié)構(gòu)的重要組成部分。分析了 LSTM型RNN的訓(xùn)練過(guò)程,包括前向傳播過(guò)程和反向傳播過(guò)程。在反向傳播過(guò)程中,激活函數(shù)及其導(dǎo)數(shù)直接影響網(wǎng)絡(luò)訓(xùn)練的收斂速度。從激活函數(shù)方面著手對(duì)循環(huán)神經(jīng)網(wǎng)絡(luò)算法進(jìn)行優(yōu)化。傳統(tǒng)的S型激活函數(shù)存在飽和區(qū)導(dǎo)致收斂速度慢,前人提出的修正線性單元避免了飽和區(qū)梯度消失問(wèn)題,但是帶來(lái)了梯度爆炸問(wèn)題。利用S型函數(shù)系數(shù)不同,非飽和區(qū)范圍不同,進(jìn)一步分析了不同系數(shù)之間的訓(xùn)練收斂速度的大小關(guān)系。通過(guò)實(shí)驗(yàn)證明了擴(kuò)展非飽和區(qū)的優(yōu)化方法有效地加快了訓(xùn)練收斂速度。從激活函數(shù)的硬件實(shí)現(xiàn)著手對(duì)循環(huán)神經(jīng)網(wǎng)絡(luò)算法進(jìn)行優(yōu)化,激活函數(shù)的硬件實(shí)現(xiàn)難度較大,具有更重要的研究意義。優(yōu)化誤差的研究,引入了擬合直線誤差修正項(xiàng),在硬件開(kāi)銷不變的前提下,誤差變?yōu)樵瓉?lái)的二分之一;優(yōu)化分割方法的研究,調(diào)整不同子區(qū)間的分割段數(shù),在硬件開(kāi)銷不變的前提下,使得誤差進(jìn)一步減小;激活函數(shù)硬件實(shí)現(xiàn)的可擴(kuò)展性研究,基于Sigmoid函數(shù)的硬件實(shí)現(xiàn),實(shí)現(xiàn)了參數(shù)化Sigmoid函數(shù)和Tanh函數(shù)。
[Abstract]:RecurrentNeuralNetworks (RecurrentNeuralNetworks, RNN) is an important branch of artificial neural network. It introduces the feedback mechanism in the hidden layer to realize the effective processing of the sequence data. Recurrent neural network has strong ability to store and process context information, and has become one of the research hotspots in speech recognition, Natural Language Processing, computer vision and other fields. On the one hand, recurrent neural network generally uses S function as activation function, while the saturation area of S function restricts the convergence speed of RNN training. Therefore, the optimization research of activation function has become a research hotspot. On the other hand, recurrent neural network mainly adopts the way of software implementation, and the research of hardware acceleration of the algorithm is of great significance. In view of the above problems and research background, the following work has been done on the basis of previous studies: the theoretical summary of recurrent neural network. The Long Short-Term Memory (LSTM) unique gate structure solves the gradient disappearance problem of the traditional recurrent neural network time dimension and becomes an important part of RNN structure. The training process of LSTM type RNN, including forward propagation process and reverse propagation process, is analyzed. In the process of backpropagation, the activation function and its derivative directly affect the convergence speed of network training. Starting from the activation function, the recurrent neural network algorithm is optimized. The traditional S activation function has saturation region, which leads to slow convergence. The corrected linear element proposed by predecessors avoids the problem of gradient disappearance in saturated region, but it brings gradient explosion problem. Using the different coefficients of the S type function and the different unsaturated zone range, the relation between the speed of training convergence between different coefficients is further analyzed. It is proved by experiments that the optimization method of the extended unsaturated zone can effectively speed up the convergence speed of training. Starting from the hardware realization of the activation function, the recurrent neural network algorithm is optimized. The hardware realization of the activation function is difficult, and it has more important research significance. Study on Optimization of error, introduced the linear fitting error correction, unchanged in hardware overhead, the error into the original 1/2; optimization of segmentation method, different sub interval segments, unchanged in hardware overhead, the error is further reduced; the activation function of the hardware implementation can extended research, realize Sigmoid function based on hardware realization of parameterized Sigmoid function and Tanh function.
【學(xué)位授予單位】：浙江大學(xué)
【學(xué)位級(jí)別】：碩士
【學(xué)位授予年份】：2017
【分類號(hào)】：TP183

【參考文獻(xiàn)】

相關(guān)期刊論文 前2條

1 高麗媛;盧達(dá);趙光宙;裘君;;應(yīng)用自動(dòng)微分的永磁同步電機(jī)預(yù)測(cè)控制[J];電機(jī)與控制學(xué)報(bào);2012年10期

2 黃聚永;袁慧梅;吳向陽(yáng);崔國(guó)亮;高琴;;基于查找表和Newton插值算法的正余弦函數(shù)的FPGA實(shí)現(xiàn)[J];繼電器;2007年16期

，

本文編號(hào)：1343190