# में नियमितीकरण के साथ त्रुटि प्रसार एल्गोरिथ्म

नमस्कार। मैं c # में मशीन लर्निंग के तरीकों को लागू करने के विषय को जारी रखना चाहता हूं, और इस लेख में मैं सीधे वितरण तंत्रिका नेटवर्क के प्रशिक्षण के लिए त्रुटि वापस प्रसार एल्गोरिदम के बारे में बात करूंगा, और सी # में इसके कार्यान्वयन को भी दे दूंगा। इस कार्यान्वयन की ख़ासियत यह है कि एल्गोरिथ्म का कार्यान्वयन उद्देश्य फ़ंक्शन (एक जिसे तंत्रिका नेटवर्क को कम करने की कोशिश कर रहा है) और न्यूरॉन्स के सक्रियण फ़ंक्शन के कार्यान्वयन से अमूर्त है। परिणाम एक प्रकार का कंस्ट्रक्टर है, जिसके साथ आप नेटवर्क के विभिन्न मापदंडों और लर्निंग एल्गोरिदम के साथ खेल सकते हैं, परिणाम देख और तुलना कर सकते हैं। यह माना जाता है कि आप पहले से ही परिचित हैं कि एक कृत्रिम तंत्रिका नेटवर्क क्या है (यदि नहीं, तो मैं दृढ़ता से अनुशंसा करता हूं कि आप पहले विकिपीडिया या इनमें से एक लेख का अध्ययन करें )। रुचि रखते हैं? हम बिल्ली के नीचे चढ़ते हैं।

पदनाम

शुरू करने के लिए, हम उस अंकन पर विचार करेंगे जिसका उपयोग मैं लेख में करूंगा, और एक के लिए हम बुनियादी अवधारणाओं को याद करेंगे, मैं न्यूरॉन्स और परतों के साथ चित्र नहीं दूंगा, यह सब विकिपीडिया और यहाँ हब पर है तो, तुरंत युद्ध में, न्यूरॉन का प्रेरित स्थानीय क्षेत्र (या सिर्फ एक योजक) ऐसा दिखता है:

• परत n के jth न्यूरॉन के भार के वेक्टर और इनपुट मानों के वेक्टर के रैखिक संयोजन का मान है
• बी - एक न्यूरॉन की पारी या विस्थापन; यदि हम सहमत हैं कि इनपुट वेक्टर के शून्य मान में हमेशा एक होता है , फिर शिफ्ट को शून्य सूचकांक के साथ वजन के रूप में नामित किया जा सकता है, और सूत्र को सरल बनाया जा सकता है
• - परत न्यूरॉन्स की संख्या n

न्यूरॉन सक्रियण फ़ंक्शन, या योजक मान का स्थानांतरण फ़ंक्शन:

• नेटवर्क के प्रत्येक न्यूरॉन का अपना सक्रियण कार्य हो सकता है
• पहले को छोड़कर सभी परतों के लिए, इनपुट वेक्टर पिछली परत का आउटपुट वेक्टर होगा, इसलिए

चलिए न्यूरॉन से ही नेटवर्क की ओर बढ़ते हैं। एक तंत्रिका नेटवर्क एक मॉडल है, इसमें पैरामीटर हैं, और लर्निंग एल्गोरिदम का कार्य त्रुटि नेटवर्क के मूल्य को कम करने के लिए ऐसे नेटवर्क मापदंडों का चयन करना है। त्रुटि फ़ंक्शन को ई द्वारा चिह्नित किया जाएगा । मॉडल के पैरामीटर न्यूरॉन्स के वजन हैं: - लेयर n के jth न्यूरॉन का वजन, जो लेयर के i-th न्यूरॉन (n - 1) में उत्पन्न होता है।

यह ग्रीक हम सीखने के एल्गोरिथ्म के हाइपरपैरमीटर को निरूपित करते हैं - सीखने की गति।

वजन में परिवर्तन डेल्टा द्वारा दर्शाया गया है :

• ग्रेडिएंट की दिशा हमें फ़ंक्शन के मान को बढ़ाने की दिशा में दिखाती है, लेकिन इसे कम करने के लिए, हमें विपरीत दिशा में जाने की आवश्यकता है

इस प्रकार, नया न्यूरॉन वजन निम्नानुसार है:
यह उल्लेखनीय है कि वजन में बदलाव के लिए नियमितीकरण को अभी भी (या बल्कि आवश्यक) जोड़ा जा सकता है। नियमितिकरण फ़ंक्शन आर मॉडल मापदंडों का एक कार्य है, हमारे मामले में, ये न्यूरॉन्स के वजन हैं। इस प्रकार, नया त्रुटि फ़ंक्शन ई + आर की तरह दिखता है, और वजन बदलने का सूत्र निम्न में परिवर्तित होता है:

• लैम्ब्डा - सीखने का हाइपरपैरामीटर, नियमितीकरण गुणांक (सीखने की गति के समान)
• मी - प्रशिक्षण नमूने का आकार

सामान्य तौर पर, नियमितीकरण के कार्यान्वयन को सीखने के एल्गोरिथ्म से अलग किया जा सकता है, लेकिन मैं इसे अभी तक नहीं करूंगा, क्योंकि सीखने के एल्गोरिथ्म का वर्तमान कार्यान्वयन वैसे भी सबसे तेज़ नहीं है, क्योंकि अन्यथा, प्रत्येक सीखने के युग में (सभी प्रशिक्षण उदाहरणों को चलाने), आपको एक चक्र में गिनना होगा। संचित त्रुटि, और दूसरे में - नियमितीकरण। एक और कारण यह है कि कई प्रकार के नियमितीकरण नहीं हैं (उदाहरण के लिए, मैं केवल एल 1 और एल 2 जानता हूं) जो कि तंत्रिका नेटवर्क को प्रशिक्षित करने में उपयोग किया जाता है। इस कार्यान्वयन में, मैं एल 2 मानक का उपयोग करूंगा, और यह सीखने के एल्गोरिथ्म का एक अभिन्न अंग होगा।

• 
• 

त्रुटि प्रचार एल्गोरिथम

सबसे पहले, प्रशिक्षण मोड पर ध्यान दें। वज़न बदलने के कई तरीके हैं:

• या प्रत्येक प्रशिक्षण उदाहरण के बाद (वास्तविक समय प्रशिक्षण, ऑनलाइन प्रशिक्षण, बैचसाइज़ = 1)
• या तो पूरे प्रशिक्षण नमूने के लिए परिवर्तन जमा करें, और फिर सभी भार (पूर्ण-बैच, बैचसाइज़ = प्रशिक्षणसेट। गति) को बदलें
• या प्रशिक्षण के उदाहरणों की एक निश्चित संख्या को चलाने के बाद (मिनी-बैच, बैचसाइज़ = any_number <trainingSet.Length)

ऑनलाइन सीखने के साथ स्थिति पर विचार करें, यह आसान होगा। तो, गति नेटवर्क इनपुट के लिए आया था नेटवर्क ने जवाब दिया हालांकि एक्स के लिए सही प्रतिक्रिया है ।
त्रुटि फ़ंक्शन E के आंशिक व्युत्पन्न पर विचार करें:

• 
  • लेकिन चूंकि त्रुटि फ़ंक्शन केवल योजक के माध्यम से व्यक्त किया जा सकता है, और योजक वजन पर निर्भर करता है, हम इस प्रविष्टि को इस प्रकार फिर से लिख सकते हैं
• 
• 
  • इस प्रकार, वजन द्वारा योजक के आंशिक व्युत्पन्न की गणना की जाती है, फिर हम वजन द्वारा उद्देश्य फ़ंक्शन के आंशिक व्युत्पन्न की अभिव्यक्ति को फिर से लिख सकते हैं; अंत में हम प्राप्त करते हैं
• 

आगे की चर्चा को दो शाखाओं में विभाजित किया गया है: अंतिम परत के लिए और शेष परतों के लिए।

आउटपुट लेयर

आउटपुट लेयर के लिए, सब कुछ सरल है, त्रुटि सुधार के लिए, हमें केवल एक वेट के अनुसार ऑब्जेक्टिव फ़ंक्शन के व्युत्पन्न की गणना करने और डेल्टा मान की गणना करने की आवश्यकता है। हम इस बात को ध्यान में रखते हैं कि उद्देश्य फ़ंक्शन पूरी तरह से केवल न्यूरॉन के आउटपुट मूल्य, या सक्रियण फ़ंक्शन के मूल्य पर निर्भर करता है, और सक्रियण फ़ंक्शन केवल योजक पर निर्भर करता है

• 

यहां यह देखा जा सकता है कि आउटपुट लेयर की त्रुटि की गणना करने के लिए, हमारे लक्ष्य फ़ंक्शन या न्यूरॉन का सक्रियण फ़ंक्शन क्या है, इसकी परवाह किए बिना, बिंदुओं पर आंशिक डेरिवेटिव के मूल्य की गणना करना आवश्यक है।

कोई छिपी हुई परत

लेकिन अगर परत आउटपुट नहीं है, तो हमें बाद की सभी परतों के त्रुटि मानों को जमा करने की आवश्यकता है।

• 
  • आगे हम इस बात का ध्यान रखते हैं कि अगली परत का योजक केवल वर्तमान परत के आउटपुट पर निर्भर करता है, और वर्तमान के आउटपुट केवल वर्तमान परत के योजक पर निर्भर करते हैं
• 
• 
  • पिछली परत के आउटपुट पर योजक के व्युत्पन्न हमें न्यूरॉन्स के बीच सिर्फ वजन देंगे, इसे देखते हुए हम मुख्य निष्कर्ष को फिर से लिखते हैं।
• 

पुनश्च: मैंने देखा है कि मैं यह दर्शाने के लिए कोष्ठक में कोष्ठक लगाना भूल गया कि यह कोई डिग्री नहीं है, बल्कि एक परत सूचकांक है; इसे ध्यान में रखें जबकि कहीं भी कोई डिग्री नहीं है।

हमारे पास क्या है:

• एक निश्चित सक्रियण फ़ंक्शन के व्युत्पन्न की गणना सिर्फ वही है जो आपको चाहिए
• अगली परत के योजक के मूल्य द्वारा उद्देश्य फ़ंक्शन के आंशिक व्युत्पन्न की गणना; यहां सब कुछ भी सरल है, क्योंकि हम अंतिम परत पर नहीं हैं, और हम पिछले से पहले तक वजन में बदलाव की गणना कर रहे हैं, इसलिए इस मूल्य की गणना पिछले चरण में पहले ही की जा चुकी है
  
  • 
  • यदि अगली परत अंतिम है, तो हम वर्तमान परत के लिए इस मान की गणना करेंगे, और इस तरह पूरे नेटवर्क के समाधान का विस्तार करेंगे
  
  

कार्यान्वयन

त्रुटि समारोह

हमने फ़ार्मुलों को समाप्त कर दिया है, आइए कार्यान्वयन के लिए आगे बढ़ें, और एक त्रुटि फ़ंक्शन की अवधारणा के साथ शुरू करें। मैंने इसे एक मीट्रिक के रूप में प्रस्तुत किया है (वास्तव में, यह मामला है)। CalculatePartialDerivaitveByV2Index विधि v2 से सूचकांक चर द्वारा इनपुट वैक्टर के लिए एक फ़ंक्शन के आंशिक व्युत्पन्न के मूल्य की गणना करता है।

public interface IMetrics<T> { double Calculate(T[] v1, T[] v2); /// <summary> /// Calculate value of partial derivative by v2[v2Index] /// </summary> T CalculatePartialDerivaitveByV2Index(T[] v1, T[] v2, int v2Index); } 

इस प्रकार, हम नेटवर्क के वास्तविक आउटपुट से अंतिम परत के लिए त्रुटि फ़ंक्शन के आंशिक व्युत्पन्न के मूल्य की गणना कर सकते हैं ।

उदाहरण के लिए, आइए कुछ कार्यान्वयन लिखते हैं।

न्यूरॉन सक्रियण समारोह

इसी तरह से, हम न्यूरॉन सक्रियण फ़ंक्शन का वर्णन करते हैं।

 public interface IFunction { double Compute(double x); double ComputeFirstDerivative(double x); } 

और उदाहरण है।

न्यूरॉन, परत और नेटवर्क

इस खंड में, हम नेटवर्क के मुख्य तत्वों की प्रस्तुति पर विचार करेंगे, मैं उनका कार्यान्वयन नहीं दूंगा, क्योंकि वह स्पष्ट है। एल्गोरिथ्म पूरी तरह से जुड़े "स्तरित" नेटवर्क के लिए दिया जाएगा, इसलिए नेटवर्क के कार्यान्वयन को तदनुसार करने की आवश्यकता होगी।

तो, न्यूरॉन निम्नानुसार है।

 public interface INeuron { /// <summary> /// Weights of the neuron /// </summary> double[] Weights { get; } /// <summary> /// Offset/bias of neuron (default is 0) /// </summary> double Bias { get; set; } /// <summary> /// Compute NET of the neuron by input vector /// </summary> /// <param name="inputVector">Input vector (must be the same dimension as was set in SetDimension)</param> /// <returns>NET of neuron</returns> double NET(double[] inputVector); /// <summary> /// Compute state of neuron /// </summary> /// <param name="inputVector">Input vector (must be the same dimension as was set in SetDimension)</param> /// <returns>State of neuron</returns> double Activate(double[] inputVector); /// <summary> /// Last calculated state in Activate /// </summary> double LastState { get; set; } /// <summary> /// Last calculated NET in NET /// </summary> double LastNET { get; set; } IList<INeuron> Childs { get; } IList<INeuron> Parents { get; } IFunction ActivationFunction { get; set; } double dEdz { get; set; } } 

क्योंकि हम पूरी तरह से जुड़े "स्तरित" नेटवर्क पर विचार कर रहे हैं, फिर चिल्ड और पेरेंट्स को लागू नहीं किया जा सकता है, लेकिन यदि आप एक सामान्य एल्गोरिथ्म बनाते हैं, तो आपको करना होगा। उन क्षेत्रों पर विचार करें जो लर्निंग एल्गोरिदम के लिए विशेष रूप से महत्वपूर्ण हैं:

• लास्टनेट - न्यूरॉन योजक, अंतिम गणना मूल्य यहां संग्रहीत है
• LastState - न्यूरॉन आउटपुट, अंतिम गणना मूल्य यहां संग्रहीत है
• dEdz न्यूरॉन का वही dE / dz है जो ऊपर उल्लिखित है, और यह गणना की जाती है कि वर्तमान न्यूरॉन किस परत पर निर्भर करता है; न्यूरॉन के योजक के संबंध में त्रुटि फ़ंक्शन का आंशिक व्युत्पन्न

नेटवर्क परत सरल दिखती है:

 public interface ILayer { /// <summary> /// Compute output of the layer /// </summary> /// <param name="inputVector">Input vector</param> /// <returns>Output vector</returns> double[] Compute(double[] inputVector); /// <summary> /// Get last output of the layer /// </summary> double[] LastOutput { get; } /// <summary> /// Get neurons of the layer /// </summary> INeuron[] Neurons { get; } /// <summary> /// Get input dimension of neurons /// </summary> int InputDimension { get; } } 

और नेटवर्क दृश्य:

 public interface INeuralNetwork { /// <summary> /// Compute output vector by input vector /// </summary> /// <param name="inputVector">Input vector (double[])</param> /// <returns>Output vector (double[])</returns> double[] ComputeOutput(double[] inputVector); Stream Save(); /// <summary> /// Train network with given inputs and outputs /// </summary> /// <param name="inputs">Set of input vectors</param> /// <param name="outputs">Set if output vectors</param> void Train(IList<DataItem<double>> data); } 

लेकिन हम एक बहुपरत तंत्रिका नेटवर्क पर विचार कर रहे हैं, इसलिए एक विशेष दृश्य का उपयोग किया जाएगा:

 public interface IMultilayerNeuralNetwork : INeuralNetwork { /// <summary> /// Get array of layers of network /// </summary> ILayer[] Layers { get; } } 

लर्निंग एल्गोरिदम

लर्निंग एल्गोरिदम को रणनीति पैटर्न के माध्यम से कार्यान्वित किया जाएगा:

 public interface ILearningStrategy<T> { /// <summary> /// Train neural network /// </summary> /// <param name="network">Neural network for training</param> /// <param name="inputs">Set of input vectors</param> /// <param name="outputs">Set of output vectors</param> void Train(T network, IList<DataItem<double>> data); } 

बेहतर समझ के लिए, मैं इस कार्यान्वयन के संदर्भ में किसी भी तंत्रिका नेटवर्क का एक विशिष्ट ट्रेन फ़ंक्शन दूंगा:

 public void Train(IList<DataItem<double>> data) { _learningStrategy.Train(this, data); } 

इनपुट प्रारूप

मैं निम्नलिखित इनपुट प्रारूप का उपयोग करता हूं:

 public class DataItem<T> { private T[] _input = null; private T[] _output = null; public DataItem() { } public DataItem(T[] input, T[] output) { _input = input; _output = output; } public T[] Input { get { return _input; } set { _input = value; } } public T[] Output { get { return _output; } set { _output = value; } } } 

जैसा कि पिछले भागों में कोड से देखा गया है, तंत्रिका नेटवर्क के साथ काम करता है

 DataItem. 
 
    
      ,         ( ),      : 
 public class LearningAlgorithmConfig { public double LearningRate { get; set; } /// <summary> /// Size of the butch. -1 means fullbutch size. /// </summary> public int BatchSize { get; set; } public double RegularizationFactor { get; set; } public int MaxEpoches { get; set; } /// <summary> /// If cumulative error for all training examples is less then MinError, then algorithm stops /// </summary> public double MinError { get; set; } /// <summary> /// If cumulative error change for all training examples is less then MinErrorChange, then algorithm stops /// </summary> public double MinErrorChange { get; set; } /// <summary> /// Function to minimize /// </summary> public IMetrics<double> ErrorFunction { get; set; } } 
 DataItem. 
 
    
      ,         ( ),      : 
 public class LearningAlgorithmConfig { public double LearningRate { get; set; } /// <summary> /// Size of the butch. -1 means fullbutch size. /// </summary> public int BatchSize { get; set; } public double RegularizationFactor { get; set; } public int MaxEpoches { get; set; } /// <summary> /// If cumulative error for all training examples is less then MinError, then algorithm stops /// </summary> public double MinError { get; set; } /// <summary> /// If cumulative error change for all training examples is less then MinErrorChange, then algorithm stops /// </summary> public double MinErrorChange { get; set; } /// <summary> /// Function to minimize /// </summary> public IMetrics<double> ErrorFunction { get; set; } } 
 DataItem. 
 
    
      ,         ( ),      : 
 public class LearningAlgorithmConfig { public double LearningRate { get; set; } /// <summary> /// Size of the butch. -1 means fullbutch size. /// </summary> public int BatchSize { get; set; } public double RegularizationFactor { get; set; } public int MaxEpoches { get; set; } /// <summary> /// If cumulative error for all training examples is less then MinError, then algorithm stops /// </summary> public double MinError { get; set; } /// <summary> /// If cumulative error change for all training examples is less then MinErrorChange, then algorithm stops /// </summary> public double MinErrorChange { get; set; } /// <summary> /// Function to minimize /// </summary> public IMetrics<double> ErrorFunction { get; set; } } 

एल्गोरिथ्म

और अंत में, पूरे संदर्भ को दिखाते हुए, हम तंत्रिका नेटवर्क लर्निंग एल्गोरिदम के वास्तविक कार्यान्वयन के लिए आगे बढ़ सकते हैं

 internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
 
      (    )   : 
 if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning..."); 
 
      ,          ,    : 
 do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); 
 
   ,        ,        .  ,  batch  ,   . 
 lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); } 
 
     ,     ,   ,   : 
 //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count); 
 
    : 
  "",           (  ,       )        dE/dz          
 //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } } 
 
           : 
       "",           (  ,       )        dE/dz,        ,              
 //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } } 
 
    ,               (  ),  : 
 //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString()); 
 
         ,    : 
 } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); 
 
  
  .      ,        ,       ,        .                .              ,       , , ,  ,        . 
     : 
  
   : 
  . 
 
 ,    ,           ,             ,    .                -) 

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)

 internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
 
      (    )   : 
 if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning..."); 
 
      ,          ,    : 
 do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); 
 
   ,        ,        .  ,  batch  ,   . 
 lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); } 
 
     ,     ,   ,   : 
 //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count); 
 
    : 
  "",           (  ,       )        dE/dz          
 //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } } 
 
           : 
       "",           (  ,       )        dE/dz,        ,              
 //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } } 
 
    ,               (  ),  : 
 //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString()); 
 
         ,    : 
 } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); 
 
  
  .      ,        ,       ,        .                .              ,       , , ,  ,        . 
     : 
  
   : 
  . 
 
 ,    ,           ,             ,    .                -) 

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)

 internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
 
      (    )   : 
 if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning..."); 
 
      ,          ,    : 
 do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); 
 
   ,        ,        .  ,  batch  ,   . 
 lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); } 
 
     ,     ,   ,   : 
 //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count); 
 
    : 
  "",           (  ,       )        dE/dz          
 //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } } 
 
           : 
       "",           (  ,       )        dE/dz,        ,              
 //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } } 
 
    ,               (  ),  : 
 //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString()); 
 
         ,    : 
 } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); 
 
  
  .      ,        ,       ,        .                .              ,       , , ,  ,        . 
     : 
  
   : 
  . 
 
 ,    ,           ,             ,    .                -) 

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)

 internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
 
      (    )   : 
 if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning..."); 
 
      ,          ,    : 
 do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); 
 
   ,        ,        .  ,  batch  ,   . 
 lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); } 
 
     ,     ,   ,   : 
 //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count); 
 
    : 
  "",           (  ,       )        dE/dz          
 //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } } 
 
           : 
       "",           (  ,       )        dE/dz,        ,              
 //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } } 
 
    ,               (  ),  : 
 //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString()); 
 
         ,    : 
 } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); 
 
  
  .      ,        ,       ,        .                .              ,       , , ,  ,        . 
     : 
  
   : 
  . 
 
 ,    ,           ,             ,    .                -) 

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)

• internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)
• internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)
• internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)
• internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)

 internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
 
      (    )   : 
 if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning..."); 
 
      ,          ,    : 
 do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); 
 
   ,        ,        .  ,  batch  ,   . 
 lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); } 
 
     ,     ,   ,   : 
 //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count); 
 
    : 
  "",           (  ,       )        dE/dz          
 //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } } 
 
           : 
       "",           (  ,       )        dE/dz,        ,              
 //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } } 
 
    ,               (  ),  : 
 //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString()); 
 
         ,    : 
 } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); 
 
  
  .      ,        ,       ,        .                .              ,       , , ,  ,        . 
     : 
  
   : 
  . 
 
 ,    ,           ,             ,    .                -) 

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)

• internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)
• internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)
• internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)
• internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)
• internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)

 internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
 
      (    )   : 
 if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning..."); 
 
      ,          ,    : 
 do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); 
 
   ,        ,        .  ,  batch  ,   . 
 lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); } 
 
     ,     ,   ,   : 
 //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count); 
 
    : 
  "",           (  ,       )        dE/dz          
 //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } } 
 
           : 
       "",           (  ,       )        dE/dz,        ,              
 //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } } 
 
    ,               (  ),  : 
 //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString()); 
 
         ,    : 
 } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); 
 
  
  .      ,        ,       ,        .                .              ,       , , ,  ,        . 
     : 
  
   : 
  . 
 
 ,    ,           ,             ,    .                -) 

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)

 internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
 
      (    )   : 
 if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning..."); 
 
      ,          ,    : 
 do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); 
 
   ,        ,        .  ,  batch  ,   . 
 lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); } 
 
     ,     ,   ,   : 
 //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count); 
 
    : 
  "",           (  ,       )        dE/dz          
 //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } } 
 
           : 
       "",           (  ,       )        dE/dz,        ,              
 //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } } 
 
    ,               (  ),  : 
 //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString()); 
 
         ,    : 
 } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); 
 
  
  .      ,        ,       ,        .                .              ,       , , ,  ,        . 
     : 
  
   : 
  . 
 
 ,    ,           ,             ,    .                -) 

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)

 internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy,  public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data). 
 
      (    )   : 
 if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning..."); 
 
      ,          ,    : 
 do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); 
 
   ,        ,        .  ,  batch  ,   . 
 lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); } 
 
     ,     ,   ,   : 
 //process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //.......................................   //hidden layers //.......................................   } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count); 
 
    : 
  "",           (  ,       )        dE/dz          
 //last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } } 
 
           : 
       "",           (  ,       )        dE/dz,        ,              
 //hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } } 
 
    ,               (  ),  : 
 //recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString()); 
 
         ,    : 
 } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange); 
 
  
  .      ,        ,       ,        .                .              ,       , , ,  ,        . 
     : 
  
   : 
  . 
 
 ,    ,           ,             ,    .                -) 

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)

internal class BackpropagationFCNLearningAlgorithm : ILearningStrategy, public void Train(IMultilayerNeuralNetwork network, IList<DataItem> data).

( ) :
if (_config.BatchSize < 1 || _config.BatchSize > data.Count) { _config.BatchSize = data.Count; } double currentError = Single.MaxValue; double lastError = 0; int epochNumber = 0; Logger.Instance.Log("Start learning...");

, , :
do { //... } while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);

, , . , batch , .
lastError = currentError; DateTime dtStart = DateTime.Now; //preparation for epoche int[] trainingIndices = new int[data.Count]; for (int i = 0; i < data.Count; i++) { trainingIndices[i] = i; } if (_config.BatchSize > 0) { trainingIndices = Shuffle(trainingIndices); }

, , , :
//process data set int currentIndex = 0; do { #region initialize accumulated error for batch, for weights and biases double[][][] nablaWeights = new double[network.Layers.Length][][]; double[][] nablaBiases = new double[network.Layers.Length][]; for (int i = 0; i < network.Layers.Length; i++) { nablaBiases[i] = new double[network.Layers[i].Neurons.Length]; nablaWeights[i] = new double[network.Layers[i].Neurons.Length][]; for (int j = 0; j < network.Layers[i].Neurons.Length; j++) { nablaBiases[i][j] = 0; nablaWeights[i][j] = new double[network.Layers[i].Neurons[j].Weights.Length]; for (int k = 0; k < network.Layers[i].Neurons[j].Weights.Length; k++) { nablaWeights[i][j][k] = 0; } } } #endregion //process one batch for (int inBatchIndex = currentIndex; inBatchIndex < currentIndex + _config.BatchSize && inBatchIndex < data.Count; inBatchIndex++) { //forward pass double[] realOutput = network.ComputeOutput(data[trainingIndices[inBatchIndex]].Input); //backward pass, error propagation //last layer //....................................... //hidden layers //....................................... } //update weights and bias for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Bias -= nablaBiases[layerIndex][neuronIndex]; for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] -= nablaWeights[layerIndex][neuronIndex][weightIndex]; } } } currentIndex += _config.BatchSize; } while (currentIndex < data.Count);

:
"", ( , ) dE/dz
//last layer for (int j = 0; j < network.Layers[network.Layers.Length - 1].Neurons.Length; j++) { network.Layers[network.Layers.Length - 1].Neurons[j].dEdz = _config.ErrorFunction.CalculatePartialDerivaitveByV2Index(data[inBatchIndex].Output, realOutput, j) * network.Layers[network.Layers.Length - 1].Neurons[j].ActivationFunction. ComputeFirstDerivative(network.Layers[network.Layers.Length - 1].Neurons[j].LastNET); nablaBiases[network.Layers.Length - 1][j] += _config.LearningRate * network.Layers[network.Layers.Length - 1].Neurons[j].dEdz; for (int i = 0; i < network.Layers[network.Layers.Length - 1].Neurons[j].Weights.Length; i++) { nablaWeights[network.Layers.Length - 1][j][i] += _config.LearningRate*(network.Layers[network.Layers.Length - 1].Neurons[j].dEdz* (network.Layers.Length > 1 ? network.Layers[network.Layers.Length - 1 - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[network.Layers.Length - 1].Neurons[j].Weights[i] / data.Count); } }

:
"", ( , ) dE/dz, ,
//hidden layers for (int hiddenLayerIndex = network.Layers.Length - 2; hiddenLayerIndex >= 0; hiddenLayerIndex--) { for (int j = 0; j < network.Layers[hiddenLayerIndex].Neurons.Length; j++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz = 0; for (int k = 0; k < network.Layers[hiddenLayerIndex + 1].Neurons.Length; k++) { network.Layers[hiddenLayerIndex].Neurons[j].dEdz += network.Layers[hiddenLayerIndex + 1].Neurons[k].Weights[j]* network.Layers[hiddenLayerIndex + 1].Neurons[k].dEdz; } network.Layers[hiddenLayerIndex].Neurons[j].dEdz *= network.Layers[hiddenLayerIndex].Neurons[j].ActivationFunction. ComputeFirstDerivative( network.Layers[hiddenLayerIndex].Neurons[j].LastNET ); nablaBiases[hiddenLayerIndex][j] += _config.LearningRate* network.Layers[hiddenLayerIndex].Neurons[j].dEdz; for (int i = 0; i < network.Layers[hiddenLayerIndex].Neurons[j].Weights.Length; i++) { nablaWeights[hiddenLayerIndex][j][i] += _config.LearningRate * ( network.Layers[hiddenLayerIndex].Neurons[j].dEdz * (hiddenLayerIndex > 0 ? network.Layers[hiddenLayerIndex - 1].Neurons[i].LastState : data[inBatchIndex].Input[i]) + _config.RegularizationFactor * network.Layers[hiddenLayerIndex].Neurons[j].Weights[i] / data.Count ); } } }

, ( ), :
//recalculating error on all data //real error currentError = 0; for (int i = 0; i < data.Count; i++) { double[] realOutput = network.ComputeOutput(data[i].Input); currentError += _config.ErrorFunction.Calculate(data[i].Output, realOutput); } currentError *= 1d/data.Count; //regularization term if (Math.Abs(_config.RegularizationFactor - 0d) > Double.Epsilon) { double reg = 0; for (int layerIndex = 0; layerIndex < network.Layers.Length; layerIndex++) { for (int neuronIndex = 0; neuronIndex < network.Layers[layerIndex].Neurons.Length; neuronIndex++) { for (int weightIndex = 0; weightIndex < network.Layers[layerIndex].Neurons[neuronIndex].Weights.Length; weightIndex++) { reg += network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex] * network.Layers[layerIndex].Neurons[neuronIndex].Weights[weightIndex]; } } } currentError += _config.RegularizationFactor * reg / (2 * data.Count); } epochNumber++; Logger.Instance.Log("Eposh #" + epochNumber.ToString() + " finished; current error is " + currentError.ToString() + "; it takes: " + (DateTime.Now - dtStart).Duration().ToString());

, :
} while (epochNumber < _config.MaxEpoches && currentError > _config.MinError && Math.Abs(currentError - lastError) > _config.MinErrorChange);


. , , , . . , , , , , .
:

:
.

, , , , . -)