Adaptive Hybrid Gradient-based Particle Swarm Optimization for Enhanced Neural Network Training
Okundalaye Oluwaseun Olumide *
Department of Mathematical Science, Adekunle Ajasin University, Akungba-Akoko, Nigeria.
*Author to whom correspondence should be addressed.
Abstract
Optimizing deep neural networks (DNNs) presents significant challenges due to complex loss landscapes, hyperparameter sensitivity, and slow convergence rates. This study introduces a Hybrid Gradient-Based Particle Swarm Optimization (HG-PSO) framework that combines the global search capability of Particle Swarm Optimization (PSO) with the local refinement efficiency of gradient-based methods. The proposed approach dynamically balances exploration and exploitation, leading to improved convergence speed, reduced overfitting, and enhanced generalization performance. Experimental evaluations using benchmark datasets—Fashion-MNIST, SVHN, and Tiny ImageNet—demonstrate that HG-PSO outperforms traditional optimizers such as Stochastic Gradient Descent (SGD), Adam, and standalone PSO. HG-PSO achieves a 15% reduction in training errors and a 12% increase in validation accuracy on average. Additionally, the method exhibits superior robustness against noisy gradients, making it well-suited for real-world deep-learning applications. These results establish HG-PSO as a powerful and efficient optimization strategy for neural network training.
Keywords: Hybrid Optimization, Particle Swarm Optimization (PSO), deep neural networks (dnns), stochastic gradient descent (sgd), adaptive learning