Development of a Hybrid Optimization Algorithm for Efficient Neural Network Training

Mohammed  Dawood Salman; Ghufran  K. Joad; Saif  Ahmed Hussein

Mohammed Dawood Salman Institute of Applied Arts, Middle Technical University, Baghdad, Iraq
Ghufran K. Joad Department of Production and Metallurgical Engineering, University of Technology, Iraq
Saif Ahmed Hussein General Directorate of Education in Salladdin, Iraq Ph.D. in Numerical Optimization

Keywords: Hestenes-Stiefel, Polak-Ribiere, hybrid algorithm, adaptive hybrid method FFNN

Abstract

In this study, a hybrid optimization algorithm in mathematical combining the Hestenes-Stiefel (HS) and Polak-Ribiere (PR) methods was developed using an adaptive approach to optimize the training of Feed-Forward Neural Networks (FFNNs). This approach aims to leverage the global convergence power of the HS method and the ability of PR to avoid local minimum. The hybrid algorithm was tested on three real world datasets (Iris, Glass, and Wine), and compared to the results obtained using the original two methods (HS and PR). The hybrid algorithm showed shorter training times across all datasets and hybrid algorithm significantly reduced the mean square error (MSE) compared to the two separate methods, resulting in faster convergence with training accuracies fir the Iris datasets, Glass dataset, and Wine dataset being 98.50%, 98.39%, and 65.98%, respectively, enhancing efficiency. The algorithm also proved to be able to handle data with high variance or nonlinearity more effectively, making it suitable for training neural networks in machine learning applications.

References

M. Benzi, “Solving large-scale linear systems: Methods and applications,” Journal of Computational and Applied Mathematics, vol. 380, 2020.

H. Zhu, “Structural optimization using conjugate gradient method with applications in engineering,” Engineering Reorts, vol. 2, no. 3, 2020.

B. Ghojogh and M. Crowley, “The theory behind overfitting, cross-validation, regularization, bagging, and boosting: Tutorial,” Machine Learning: Science and Technology, vol. 2, 2020.

Fletcher, R. (2013). Practical methods of optimization. John Wiley & Sons.

H. Zhu and X. Pan, “Structural optimization using conjugate gradient method with applications in engineering,” Engineering Reports, vol. 2, no. 3, 2020.

J. Liu, H. Zhang, and Y. Zhou, “A parallel algorithm for solving large linear systems in optimization,” Optimization Methods and Software, vol. 35, no. 6, pp. 1133–1155, 2020.

B. Ghojogh and M. Crowley, “The theory behind overfitting, cross-validation, regularization, bagging, and boosting: A tutorial,” Machine Learning: Science and Technology, vol. 2, no. 1, 2020.

M. S. Jameel and Z. M. Abdullah, “A new shifted conjugate gradient method based on shifted quasi-Newton condition,” Journal of Physics: Conference Series, vol. 1818, no. 1, 2021.

L. Zhang and W. Zhou, “Two descent hybrid conjugate gradient methods for optimization,” J. Comput. Appl. Math., vol. 216, no. 1, pp. 251–264, 2008.

W. W. Hager and H. Zhang, “A survey of nonlinear conjugate gradient methods,” Pacific Journal of Optimization, vol. 2, no. 1, pp. 35–58, 2020.

M. Al-Baali, “Improved convergence properties of the Fletcher-Reeves conjugate gradient method,” IMA Journal of Numerical Analysis, vol. 5, no. 1, pp. 121–124, 2021.

A. Al-Saidi, “Improved Fletcher-Reeves methods based on new scaling techniques,” Sultan Qaboos Univ. J. Sci. [SQUJS], vol. 26, no. 2, pp. 141–151, 2021.

Z. M. Abdullah, H. M. Khudhur, and A. Khairulla Ahmed, “Modification of the new conjugate gradient algorithm to solve nonlinear fuzzy equations,” Indones. J. Electr. Eng. Comput. Sci., vol. 27, no. 3, p. 1525, 2022.

M. Azzam and N. Hawraz, “Four-Term Conjugate Gradient (CG) Method Based on Pure Conjugacy Condition for Unconstrained Optimization,” Kirkuk Journal of Science, vol. 13, pp. 101–113, 2018.

H. Mohammed and K. K. Abbo, “New hybrid of Conjugate Gradient Technique for Solving Fuzzy Nonlinear Equations,” Journal of Soft Computing and Artificial Intelligence, vol. 2, no. 1, pp. 1–8, 2021.

K. K. Abbo and H. M. Khudhur, “New A hybrid conjugate gradient Fletcher-Reeves and Polak-Ribiere algorithm for unconstrained optimization,” Tikrit J. Pure Sci., vol. 21, no. 1, pp. 124–129, 2023.

H. M. Khudhur and K. K. Abbo, “A new type of Conjugate Gradient Technique for solving fuzzy nonlinear algebraic equations,” J. Phys. Conf. Ser., vol. 1879, no. 2, p. 022111, 2021.

H. N. Jabbar, K. Khalil, and H. M. Abbo, “Four--term conjugate gradient (CG) method based on pure conjugacy condition for unconstrained optimization,” Univ. J. Sci. Stud, vol. 13, pp. 101–113, 2018.

Y. A. Laylani, K. Khalil, and H. M. Abbo, “Training feed forward neural network with modified Fletcher-Reeves method,” Journal of Multidisciplinary Modeling and Optimization, vol. 1, pp. 14–22, 2018.

K. K. Abbo and F. H. Mohammed, “Spectral Fletcher-Reeves Algorithm for Solving Non-Linear Unconstrained Optimization Problems,” Iraqi Journal of Statistical Sciences, vol. 19, pp. 21–38, 2011.

T. Gao, Z. Zhang, Q. Chang, X. Xie, P. Ren, and J. Wang, “Conjugate gradient-based Takagi-Sugeno fuzzy neural network parameter identification and its convergence analysis,” Neurocomputing, vol. 364, pp. 168–181, 2019.

F. Venturi, “How to choose a glass: the influence of glass parameters on the profile of different wines (full bodied red and rosé) during tasting,” in ISPROF 2015 Book of Abstracts 2nd International Symposium on Profiling, Proteomass, 2015.

L. Omelina, J. Goga, J. Pavlovicova, M. Oravec, and B. Jansen, “A survey of iris datasets,” Image Vis. Comput., vol. 108, no. 104109, p. 104109, 2021.