Pi-Sigma和Sigma-Pi-Sigma神經(jīng)網(wǎng)絡的正則化方法
發(fā)布時間：2020-10-30 23:47
在最近幾年,神經(jīng)網(wǎng)絡已經(jīng)被廣泛的應用于各種回歸和分類問題。通過將正則項加入到神經(jīng)網(wǎng)絡的學習過程中,研究者提出了許多正則化技術來處理與神經(jīng)網(wǎng)絡相關的問題。其中,兩種經(jīng)典的正則項(懲罰項)分別是運用L2范數(shù)和運用L1或L1/2范數(shù)。L2范數(shù)的功能主要是獲得有界的網(wǎng)絡權值并提高網(wǎng)絡的泛化能力。而L1或L1/2范數(shù)的功能主要是使網(wǎng)絡具有稀疏性,以便減少神經(jīng)網(wǎng)絡使用的節(jié)點和權值,與此同時并不引起對網(wǎng)絡效率的破壞。本文考慮高階神經(jīng)網(wǎng)絡(HONNs)的正則化方法。研究者已經(jīng)證明,HONNs在許多方面比普通的一階網(wǎng)絡更高效。從一些方面來講,稀疏化對HONNs更重要,因為通常情況下HONNs中有更多的節(jié)點和權值。特別地,本文考慮pi-sigma神經(jīng)網(wǎng)絡(PSNNs)和 sigma-pi-sigma 神經(jīng)網(wǎng)絡(SPSNNs)。本論文的主要內(nèi)容如下:1.在第2章,本文研究用于PSNNs的帶L2內(nèi)懲罰的在線梯度算法。這里,L2范數(shù)是關于每個Pi節(jié)點的輸入值的。證明了誤差函數(shù)的單調(diào)性、權值的有界性以及弱收斂定理和強收斂定理。2.在第3章,本文描述了另一種用于PSNNs的L2內(nèi)懲罰。不同于第2章,本章中L2范數(shù)是關于網(wǎng)絡中每一個權值的。證明了批處理梯度算法的收斂性。并證明了在訓練迭代中帶懲罰項的誤差函數(shù)的單調(diào)性,以及權值序列的一致有界性。將該算法應用于求解四維奇偶問題和Gabor函數(shù)問題以支持我們的理論結(jié)果。3.在第4章,提出了一個帶光滑L1/2正則化的離線梯度法來訓練和修剪PSNNs。因為涉及絕對值函數(shù),原始L1/2正則項在原點不光滑。這會導致計算中出現(xiàn)振蕩現(xiàn)象,并且非常難于進行收斂性分析。本文提出了使用光滑函數(shù)代替并近似絕對值函數(shù),得到一個PSNNs的光滑L1/2正則化方法。數(shù)值模擬表明,光滑L1/2正則化方法消除了計算中的振蕩,得到更好的學習準確率。我們也能夠證明所提出的學習方法的收斂性定理。4.在第5章,本文考慮更重要的Sigma-Pi-Sigma神經(jīng)網(wǎng)絡(SPSNNs)。在現(xiàn)有文獻中,為了減少Pi層中Pi節(jié)點的個數(shù),在SPSNNs中,研究者采用了一種特殊的多項式Ps。當令其他的變量都是常數(shù)時,Ps中每個多項式關于每一個特定的變量σi都是線性的。這種選擇可能是直觀的,但未必是最好的。本文提出了一種自適應的方法來尋找一個給定問題的更好的多項式。為了闡明提出的方法,本文從一個確定階數(shù)的完整多項式出發(fā)。然后,在學習過程中對給定問題,采用正則化技術減少所需多項式的數(shù)目,最終得到一個新的SPSNN,其所用的多項式的數(shù)量(=Pi層中的節(jié)點數(shù))和Ps中多項式的數(shù)量相同。一些基準問題的數(shù)值實驗表明,新的SPSNN表現(xiàn)比帶多項式Ps的傳統(tǒng)SPSNN好。
【學位單位】：大連理工大學
【學位級別】：博士
【學位年份】：2018
【中圖分類】：TP183
【文章目錄】：
ABSTRACT
摘要
1 Introduction
1.1 History of Artificial Neural Networks
1.2 Components of Artificial Neural Networks
1.3 Artificial Neurons
1.4 Components of an Artificial Neuron
1.4.1 Weights
1.4.2 Activation Functions
1.4.3 Bias
1.4.4 Training of Neural Network
1.5 A Model of High-Order Neural Networks
1.5.1 Sigma-Pi Neural Networks
1.5.2 Pi-Sigma Neural Networks
1.5.3 Sigma-Pi-Sigma Neural Networks
1.6 Regularization Method
1.7 Objectives and Scope of the Study
2 Convergence of Online Gradient Method for Pi-Sigma Neural Networks with Inner-Penalty Terms
2.1 Pi-Sigma Neural Network with Inner-Penalty Algorithm
2.2 Preliminary Lemmas
2.3 Convergence Theorems
3 Batch Gradient Method for Training of Pi-Sigma Neural Network with Penalty
3.1 Batch Gradient Method with Penalty Term
3.2 Main Results
3.3 Simulation Results
3.3.1 Parity Problem
3.3.2 Function Regression Problem
3.4 Proofs
4 A Modified Higher-Order Feedforward Neural Network with Smoothing Regularization
1/2 Regularization'> 4.1 Offline Gradient Method with Smoothing L1/2 Regularization
1/2 Regularization'> 4.1.1 Error Function with L1/2 Regularization
1/2 Regularization'> 4.1.2 Error Function with Smoothing L1/2 Regularization
4.2 Main Results
4.3 Numerical Experiments
4.3.1 Classification Problems
4.3.2 Approximation of Gabor Function
4.3.3 Approximation of Mayas Function
4.4 Proofs
5 Choice of Multinomials for Sigma-Pi-Sigma Neural Networks
5.1 Introduction
5.2 Description of the Proposed Method
5.2.1 Network Structure
1/2 Regularization'> 5.2.2 Error Function with L1/2 Regularization
1/2 Regularization'> 5.2.3 Error Function with Smoothing L1/2 Regularization
5.3 Algorithm
5.4 Numerical Experiments
5.4.1 Mayas' Function Approximate
5.4.2 Gabor Function Approximate
5.4.3 Sonar Data Classification
5.4.4 Pima Indians Diabetes Data Classification
6 Summary and Further Prospect
6.1 Conclusion
6.2 Innovation Points
6.3 Further Studies
References
Published Academic Articles during PhD period
Acknowledgements
Author Introduction
【參考文獻】
本文編號：2863168
【學位單位】：大連理工大學
【學位級別】：博士
【學位年份】：2018
【中圖分類】：TP183
【文章目錄】：
ABSTRACT
摘要
1 Introduction
1.1 History of Artificial Neural Networks
1.2 Components of Artificial Neural Networks
1.3 Artificial Neurons
1.4 Components of an Artificial Neuron
1.4.1 Weights
1.4.2 Activation Functions
1.4.3 Bias
1.4.4 Training of Neural Network
1.5 A Model of High-Order Neural Networks
1.5.1 Sigma-Pi Neural Networks
1.5.2 Pi-Sigma Neural Networks
1.5.3 Sigma-Pi-Sigma Neural Networks
1.6 Regularization Method
1.7 Objectives and Scope of the Study
2 Convergence of Online Gradient Method for Pi-Sigma Neural Networks with Inner-Penalty Terms
2.1 Pi-Sigma Neural Network with Inner-Penalty Algorithm
2.2 Preliminary Lemmas
2.3 Convergence Theorems
3 Batch Gradient Method for Training of Pi-Sigma Neural Network with Penalty
3.1 Batch Gradient Method with Penalty Term
3.2 Main Results
3.3 Simulation Results
3.3.1 Parity Problem
3.3.2 Function Regression Problem
3.4 Proofs
4 A Modified Higher-Order Feedforward Neural Network with Smoothing Regularization
1/2 Regularization'> 4.1 Offline Gradient Method with Smoothing L1/2 Regularization
1/2 Regularization'> 4.1.1 Error Function with L1/2 Regularization
1/2 Regularization'> 4.1.2 Error Function with Smoothing L1/2 Regularization
4.2 Main Results
4.3 Numerical Experiments
4.3.1 Classification Problems
4.3.2 Approximation of Gabor Function
4.3.3 Approximation of Mayas Function
4.4 Proofs
5 Choice of Multinomials for Sigma-Pi-Sigma Neural Networks
5.1 Introduction
5.2 Description of the Proposed Method
5.2.1 Network Structure
1/2 Regularization'> 5.2.2 Error Function with L1/2 Regularization
1/2 Regularization'> 5.2.3 Error Function with Smoothing L1/2 Regularization
5.3 Algorithm
5.4 Numerical Experiments
5.4.1 Mayas' Function Approximate
5.4.2 Gabor Function Approximate
5.4.3 Sonar Data Classification
5.4.4 Pima Indians Diabetes Data Classification
6 Summary and Further Prospect
6.1 Conclusion
6.2 Innovation Points
6.3 Further Studies
References
Published Academic Articles during PhD period
Acknowledgements
Author Introduction
【參考文獻】
相關期刊論文 前3條
1 徐宗本;郭海亮;王堯;張海;;L_(1/2)正則子在L_q(0<q<1)正則子中的代表性:基于相位圖的實驗研究(英文)[J];自動化學報;2012年07期
2 ;L_(1/2) regularization[J];Science China(Information Sciences);2010年06期
3 孔俊 ,吳微;Online Gradient Methods with a Punishing Term for Neural Networks[J];Northeastern Mathematical Journal;2001年03期
本文編號：2863168
本文鏈接：http://sikaile.net/kejilunwen/zidonghuakongzhilunwen/2863168.html
最近更新
教材專著
