在最近幾年,神經(jīng)網(wǎng)絡(luò)已經(jīng)被廣泛的應(yīng)用于各種回歸和分類問(wèn)題。通過(guò)將正則項(xiàng)加入到神經(jīng)網(wǎng)絡(luò)的學(xué)習(xí)過(guò)程中,研究者提出了許多正則化技術(shù)來(lái)處理與神經(jīng)網(wǎng)絡(luò)相關(guān)的問(wèn)題。其中,兩種經(jīng)典的正則項(xiàng)(懲罰項(xiàng))分別是運(yùn)用L2范數(shù)和運(yùn)用L1或L1/2范數(shù)。L2范數(shù)的功能主要是獲得有界的網(wǎng)絡(luò)權(quán)值并提高網(wǎng)絡(luò)的泛化能力。而L1或L1/2范數(shù)的功能主要是使網(wǎng)絡(luò)具有稀疏性,以便減少神經(jīng)網(wǎng)絡(luò)使用的節(jié)點(diǎn)和權(quán)值,與此同時(shí)并不引起對(duì)網(wǎng)絡(luò)效率的破壞。本文考慮高階神經(jīng)網(wǎng)絡(luò)(HONNs)的正則化方法。研究者已經(jīng)證明,HONNs在許多方面比普通的一階網(wǎng)絡(luò)更高效。從一些方面來(lái)講,稀疏化對(duì)HONNs更重要,因?yàn)橥ǔＧ闆r下HONNs中有更多的節(jié)點(diǎn)和權(quán)值。特別地,本文考慮pi-sigma神經(jīng)網(wǎng)絡(luò)(PSNNs)和 sigma-pi-sigma 神經(jīng)網(wǎng)絡(luò)(SPSNNs)。本論文的主要內(nèi)容如下:1.在第2章,本文研究用于PSNNs的帶L2內(nèi)懲罰的在線梯度算法。這里,L2范數(shù)是關(guān)于每個(gè)Pi節(jié)點(diǎn)的輸入值的。證明了誤差函數(shù)的單調(diào)性、權(quán)值的有界性以及弱收斂定理和強(qiáng)收斂定理。2.在第3章,本文描述了另一種用于PSNNs的L2內(nèi)懲罰。不同于第2章,本章中L2范數(shù)是關(guān)于網(wǎng)絡(luò)中每一個(gè)權(quán)值的。證明了批處理梯度算法的收斂性。并證明了在訓(xùn)練迭代中帶懲罰項(xiàng)的誤差函數(shù)的單調(diào)性,以及權(quán)值序列的一致有界性。將該算法應(yīng)用于求解四維奇偶問(wèn)題和Gabor函數(shù)問(wèn)題以支持我們的理論結(jié)果。3.在第4章,提出了一個(gè)帶光滑L1/2正則化的離線梯度法來(lái)訓(xùn)練和修剪PSNNs。因?yàn)樯婕敖^對(duì)值函數(shù),原始L1/2正則項(xiàng)在原點(diǎn)不光滑。這會(huì)導(dǎo)致計(jì)算中出現(xiàn)振蕩現(xiàn)象,并且非常難于進(jìn)行收斂性分析。本文提出了使用光滑函數(shù)代替并近似絕對(duì)值函數(shù),得到一個(gè)PSNNs的光滑L1/2正則化方法。數(shù)值模擬表明,光滑L1/2正則化方法消除了計(jì)算中的振蕩,得到更好的學(xué)習(xí)準(zhǔn)確率。我們也能夠證明所提出的學(xué)習(xí)方法的收斂性定理。4.在第5章,本文考慮更重要的Sigma-Pi-Sigma神經(jīng)網(wǎng)絡(luò)(SPSNNs)。在現(xiàn)有文獻(xiàn)中,為了減少Pi層中Pi節(jié)點(diǎn)的個(gè)數(shù),在SPSNNs中,研究者采用了一種特殊的多項(xiàng)式Ps。當(dāng)令其他的變量都是常數(shù)時(shí),Ps中每個(gè)多項(xiàng)式關(guān)于每一個(gè)特定的變量σi都是線性的。這種選擇可能是直觀的,但未必是最好的。本文提出了一種自適應(yīng)的方法來(lái)尋找一個(gè)給定問(wèn)題的更好的多項(xiàng)式。為了闡明提出的方法,本文從一個(gè)確定階數(shù)的完整多項(xiàng)式出發(fā)。然后,在學(xué)習(xí)過(guò)程中對(duì)給定問(wèn)題,采用正則化技術(shù)減少所需多項(xiàng)式的數(shù)目,最終得到一個(gè)新的SPSNN,其所用的多項(xiàng)式的數(shù)量(=Pi層中的節(jié)點(diǎn)數(shù))和Ps中多項(xiàng)式的數(shù)量相同。一些基準(zhǔn)問(wèn)題的數(shù)值實(shí)驗(yàn)表明,新的SPSNN表現(xiàn)比帶多項(xiàng)式Ps的傳統(tǒng)SPSNN好。
【學(xué)位單位】：大連理工大學(xué)
【學(xué)位級(jí)別】：博士
【學(xué)位年份】：2018
【中圖分類】：TP183
【文章目錄】：
ABSTRACT
摘要
1 Introduction
    1.1 History of Artificial Neural Networks
    1.2 Components of Artificial Neural Networks
    1.3 Artificial Neurons
    1.4 Components of an Artificial Neuron
        1.4.1 Weights
        1.4.2 Activation Functions
        1.4.3 Bias
        1.4.4 Training of Neural Network
    1.5 A Model of High-Order Neural Networks
        1.5.1 Sigma-Pi Neural Networks
        1.5.2 Pi-Sigma Neural Networks
        1.5.3 Sigma-Pi-Sigma Neural Networks
    1.6 Regularization Method
    1.7 Objectives and Scope of the Study
2 Convergence of Online Gradient Method for Pi-Sigma Neural Networks with Inner-Penalty Terms
    2.1 Pi-Sigma Neural Network with Inner-Penalty Algorithm
    2.2 Preliminary Lemmas
    2.3 Convergence Theorems
3 Batch Gradient Method for Training of Pi-Sigma Neural Network with Penalty
    3.1 Batch Gradient Method with Penalty Term
    3.2 Main Results
    3.3 Simulation Results
        3.3.1 Parity Problem
        3.3.2 Function Regression Problem
    3.4 Proofs
4 A Modified Higher-Order Feedforward Neural Network with Smoothing Regularization
1/2 Regularization'>    4.1 Offline Gradient Method with Smoothing L_1/2 Regularization
1/2 Regularization'>        4.1.1 Error Function with L_1/2 Regularization
1/2 Regularization'>        4.1.2 Error Function with Smoothing L_1/2 Regularization
    4.2 Main Results
    4.3 Numerical Experiments
        4.3.1 Classification Problems
        4.3.2 Approximation of Gabor Function
        4.3.3 Approximation of Mayas Function
    4.4 Proofs
5 Choice of Multinomials for Sigma-Pi-Sigma Neural Networks
    5.1 Introduction
    5.2 Description of the Proposed Method
        5.2.1 Network Structure
1/2 Regularization'>        5.2.2 Error Function with L_1/2 Regularization
1/2 Regularization'>        5.2.3 Error Function with Smoothing L_1/2 Regularization
    5.3 Algorithm
    5.4 Numerical Experiments
        5.4.1 Mayas' Function Approximate
        5.4.2 Gabor Function Approximate
        5.4.3 Sonar Data Classification
        5.4.4 Pima Indians Diabetes Data Classification
6 Summary and Further Prospect
    6.1 Conclusion
    6.2 Innovation Points
    6.3 Further Studies
References
Published Academic Articles during PhD period
Acknowledgements
Author Introduction

【參考文獻(xiàn)】

相關(guān)期刊論文 前3條

1 徐宗本;郭海亮;王堯;張海;;L_(1/2)正則子在L_q(0<q<1)正則子中的代表性:基于相位圖的實(shí)驗(yàn)研究(英文)[J];自動(dòng)化學(xué)報(bào);2012年07期

2 ;L_(1/2) regularization[J];Science China(Information Sciences);2010年06期

3 孔俊 ,吳微;Online Gradient Methods with a Punishing Term for Neural Networks[J];Northeastern Mathematical Journal;2001年03期

本文編號(hào)：2863168