Fast grid search: A grid search-inspired algorithm for optimizing hyperparameters of support vector regression

Açıkkar, Mustafa

doi:10.55730/1300-0632.4056

Fast grid search: A grid search-inspired algorithm for optimizing hyperparameters of support vector regression

Mustafa AÇIKKAR (Adana Alparslan Türkeş Bilim ve Teknoloji Üniversitesi, Bilgisayar ve Bilişim Mühendisliği Fakültesi, Yazılım Mühendisliği Bölümü, Adana, Türkiye)

Turkish Journal of Electrical Engineering and Computer Sciences

34 0

Yıl: 2024 Cilt: 32 Sayı: 1 Sayfa Aralığı: 68 - 92 Metin Dili: İngilizce DOI: 10.55730/1300-0632.4056 İndeks Tarihi: 14-03-2024

Fast grid search: A grid search-inspired algorithm for optimizing hyperparameters of support vector regression

Öz:

This study presents a fast hyperparameter optimization algorithm based on the benefits and shortcomings of the standard grid search (GS) algorithm for support vector regression (SVR). This presented GS-inspired algorithm, called fast grid search (FGS), was tested on benchmark datasets, and the impact of FGS on prediction accuracy was primarily compared with the GS algorithm on which it is based. To validate the eﬀicacy of the proposed algorithm and conduct a comprehensive comparison, two additional hyperparameter optimization techniques, namely particle swarm optimization and Bayesian optimization, were also employed in the development of models on the given datasets. The evaluation of the models’ predictive performance was conducted by assessing root mean square error, mean absolute error, and mean absolute percentage error. In addition to these metrics, the number of evaluated submodels and the time required for optimization were used as determinative performance measures of the presented models. Experimental results proved that the FGS-optimized SVR models yield precise performance, supporting the reliability, validity, and applicability of the FGS algorithm. As a result, the FGS algorithm can be offered as a faster alternative in optimizing the hyperparameters of SVR in terms of execution time.

Anahtar Kelime: Fast grid search support vector regression hyperparameter optimization grid search

Belge Türü: Makale Makale Türü: Araştırma Makalesi Erişim Türü: Erişime Açık

[1] Tsirikoglou P, Abraham S, Contino F, Lacor C, Ghorbaniasl G. A hyperparameters selection technique for support vector regression models. Applied Soft Computing 2017; 61: 139-148. https://doi.org/10.1016/j.asoc.2017.07.017
[2] Yoo KH, Back JH, Na MG, Kim JH, Hur S et al. Prediction of golden time using SVR for recovering SIS under severe accidents. Annals of Nuclear Energy 2016; 94: 102-108. https://doi.org/10.1016/J.ANUCENE.2016.02.029
[3] Vapnik V. The Nature of Statistical Learning Theory. New York, NY, USA: Springer, 1995.
[4] Drucker H, Burges CJC, Kaufman L, Smola A, Vapnik V. Support vector regression machines. Advances in Neural Information Processing Systems 1997; 28 (7): 779-784.
[5] Ayodeji A, Liu Y. SVR optimization with soft computing algorithms for incipient SGTR diagnosis. Annals of Nuclear Energy 2018; 121: 89-100. https://doi.org/10.1016/J.ANUCENE.2018.07.011
[6] Terrault NA, Hassanein TI. Management of the patient with SVR. Journal of Hepatology 2016; 65 (1): 120-129. https://doi.org/10.1016/J.JHEP.2016.08.001
[7] Oyehan TA, Alade IO, Bagudu A, Sulaiman KO, Olatunji SO et al. Predicting of the refractive index of haemoglobin using the hybrid GA-SVR approach. Computers in Biology Medicine 2018; 98: 85-92. https://doi.org/10.1016/j.compbiomed.2018.04.024
[8] Boegli M, Stauffer Y. SVR based PV models for MPC based energy flow management. Energy Procedia 2017; 122: 133-138. https://doi.org/10.1016/J.EGYPRO.2017.07.317
[9] Jebadurai J, Peter JD. SK-SVR: Sigmoid kernel support vector regression based in-scale single image super- resolution. Pattern Recognition Letters 2017; 94: 144-153. https://doi.org/10.1016/J.PATREC.2017.04.013
[10] Zhang WY, Hong WC, Dong Y, Tsai G, Sung JT et al. Application of SVR with chaotic GASA algorithm in cyclic electric load forecasting. Energy 2012; 45 (1): 850-858. https://doi.org/10.1016/j.energy.2012.07.006
[11] Zhang X, Chen X, He Z. An ACO-based algorithm for parameter optimization of support vector machines. Expert Systems with Applications 2010; 37 (9): 6618-6628. https://doi.org/10.1016/J.ESWA.2010.03.067
[12] Wei J, Zhang R, Yu Z, Hu R, Tang J et al. A BPSO-SVM algorithm based on memory renewal and enhanced mutation mechanisms for feature selection. Applied Soft Computing 2017; 58: 176-192. https://doi.org/10.1016/J.ASOC.2017.04.061
[13] Alam MS, Sultana N, Hossain SMZ. Bayesian optimization algorithm based support vector regression analysis for estimation of shear capacity of FRP reinforced concrete members. Applied Soft Computing 2021; 105: 107281. https://doi.org/10.1016/j.asoc.2021.107281
[14] Elsayad AM, Nassef AM, Al-Dhaifallah M. Bayesian optimization of multiclass SVM for eﬀicient diagno- sis of erythemato-squamous diseases. Biomedical Signal Processing and Control 2022; 71 (Part B): 103223. https://doi.org/10.1016/j.bspc.2021.103223
[15] Sun Y, Ding S, Zhang Z, Jia W. An improved grid search algorithm to optimize SVR for prediction. Soft Computing 2021; 25: 5633-5644. https://doi.org/10.1007/s00500-020-05560-w
[16] Huang CL, Dun JF. A distributed PSO-SVM hybrid system with feature selection and parameter optimization. Applied Soft Computing 2008; 8 (4): 1381-1391. https://doi.org/10.1016/J.ASOC.2007.10.007
[17] Harish N, Mandal S, Rao S, Patil SG. Particle swarm optimization based support vector machine for damage level prediction of non-reshaped berm breakwater. Applied Soft Computing 2015; 27: 313-321. https://doi.org/10.1016/J.ASOC.2014.10.041
[18] Lin SW, Ying KC, Chen SC, Lee ZJ. Particle swarm optimization for parameter determination and fea- ture selection of support vector machines. Expert Systems with Applications 2008; 35 (4): 1817-1824. https://doi.org/10.1016/J.ESWA.2007.08.088
[19] Narayanan G, Joshi M, Dutta P, Kalita K. PSO-tuned support vector machine metamodels for assessment of turbulent flows in pipe bends. Engineering Computations 2020; 37 (3): 981-1001. https://doi.org/10.1108/EC-05- 2019-0244
[20] Li Z, Zhang J, Liu T, Wang Y, Pei J et al. Using PSO-SVR algorithm to predict asphalt pavement performance. Journal of Performance of Constructed Facilities 2021; 35 (6): 04021094. https://doi.org/10.1061/(ASCE)CF.1943- 5509.0001666
[21] Chu XL, Zhao RJ. A building carbon emission prediction model by PSO-SVR method under multi-criteria evalua- tion. Journal of Intelligent & Fuzzy Systems 2021; 41 (6): 7473-7484. https://doi.org/10.3233/JIFS-211435
[22] Wang J, Wang X, Li X, Yi J. A hybrid particle swarm optimization algorithm with dynamic adjustment of inertia weight based on a new feature selection method to optimize SVM parameters. Entropy 2023; 25 (3): 531. https://doi.org/10.3390/e25030531
[23] Kouziokas GN. A new W-SVM kernel combining PSO-neural network transformed vector and Bayesian op- timized SVM in GDP forecasting. Engineering Applications of Artificial Intelligence 2020; 92: 103650. https://doi.org/10.1016/j.engappai.2020.103650
[24] Sultana N, Hossain SMZ, Abusaad M, Alanbar N, Senan Y et al. Prediction of biodiesel production from mi- croalgal oil using Bayesian optimization algorithm-based machine learning approaches. Fuel 2022; 309: 122184. https://doi.org/10.1016/j.fuel.2021.122184
[25] Chen KY, Wang CH. Support vector regression with genetic algorithms in forecasting tourism demand. Tourism Management 2007; 28 (1): 215-226. https://doi.org/10.1016/j.tourman.2005.12.018
[26] Jubert de Almeida B, Ferreira Neves R, Horta N. Combining support vector machine with genetic algorithms to optimize investments in Forex markets with high leverage. Applied Soft Computing 2018; 64: 596-613. https://doi.org/10.1016/J.ASOC.2017.12.047
[27] Chen KY. Forecasting systems reliability based on support vector regression with genetic algorithms. Reliability Engineering & System Safety 2007; 92 (4): 423-432. https://doi.org/10.1016/J.RESS.2005.12.014
[28] Zhan A, Du F, Chen Z, Yin G, Wang M et al. A traﬀic flow forecasting method based on the GA-SVR. Journal of High Speed Networks 2022; 28: 97-106. https://doi.org/10.3233/JHS-220682
[29] Luo Z, Hasanipanah M, Bakhshandeh Amnieh H, Brindhadevi K, Tahir MM. GA-SVR: A novel hybrid data- driven model to simulate vertical load capacity of driven piles. Engineering with Computers 2021; 37: 823-831. https://doi.org/10.1007/s00366-019-00858-2
[30] Zhang A, Yu D, Zhang Z. TLSCA-SVM fault diagnosis optimization method based on transfer learning. Processes 2022; 10 (2): 362. https://doi.org/10.3390/pr10020362
[31] Liu G, Jia W, Wang M, Heidari AA, Chen H et al. Predicting cervical hyperextension injury: a covariance guided sine cosine support vector machine. IEEE Access 2020; 8: 46895-46908. https://doi.org/10.1109/ACCESS.2020.2978102
[32] Li S, Fang H, Liu X. Parameter optimization of support vector regression based on sine cosine algorithm. Expert Systems with Applications 2018; 91: 63-77. https://doi.org/10.1016/j.eswa.2017.08.038
[33] Pan M, Li C, Gao R, Huang Y, You H et al. Photovoltaic power forecasting based on a support vec- tor machine with improved ant colony optimization. Journal of Cleaner Production 2020; 277: 123948. https://doi.org/10.1016/j.jclepro.2020.123948
[34] Raji M, Tahroudi MN, Ye F, Dutta J. Prediction of heterogeneous Fenton process in treatment of melanoidin- containing wastewater using data-based models. Journal of Environmental Management 2022; 307: 114518. https://doi.org/10.1016/j.jenvman.2022.114518
[35] Ahmed HU, Mostafa RR, Mohammed A, Sihag P, Qadir A. Support vector regression (SVR) and grey wolf optimization (GWO) to predict the compressive strength of GGBFS-based geopolymer concrete. Neural Computing and Applications 2023; 35: 2909-2926. https://doi.org/10.1007/s00521-022-07724-1
[36] Wu W, Zhang M, Zhao L, Dong H, Zhang J. CFD-DPM data-driven GWO-SVR for fast prediction of nitrate decomposition in blast furnaces with nozzle arrangement optimization. Process Safety and Environmental Protection 2023; 176: 438-449. https://doi.org/10.1016/j.psep.2023.06.029
[37] Fan C, Zheng Y, Wang S, Ma J. Prediction of bond strength of reinforced concrete structures based on feature selection and GWO-SVR model. Construction and Building Materials 2023; 400: 132602. https://doi.org/10.1016/j.conbuildmat.2023.132602
[38] Gauthama Raman MR, Somu N, Jagarapu S, Manghnani T, Selvam T et al. An eﬀicient intrusion detection tech- nique based on support vector machine and improved binary gravitational search algorithm. Artificial Intelligence Review 2020; 53 (5): 3255-3286. https://doi.org/10.1007/s10462-019-09762-z
[39] Alrefaee S, Al Bakal S, Algamal Z. Hyperparameters optimization of support vector regression using black hole algorithm. International Journal of Nonlinear Analysis and Applications 2022; 13 (1): 3441-3450. https://doi.org/10.22075/ijnaa.2022.6107
[40] Huang Q, Wang C, Ye Y, Wang L, Xie N. Recognition of EEG based on improved black widow algorithm optimized SVM. Biomedical Signal Processing and Control 2023; 81: 104454. https://doi.org/10.1016/j.bspc.2022.104454
[41] Algamal ZY, Qasim MK, Lee MH, Ali TTM. Improving grasshopper optimization algorithm for hyperparameters estimation and feature selection in support vector regression. Chemometrics and Intelligent Laboratory Systems 2021; 208: 104196. https://doi.org/10.1016/j.chemolab.2020.104196
[42] Faris H, Hassonah MA, Al-Zoubi AM, Mirjalili S, Aljarah I. A multi-verse optimizer approach for feature selection and optimizing SVM parameters based on a robust system architecture. Neural Computing & Applications 2018; 30 (8): 2355-2369. https://doi.org/10.1007/s00521-016-2818-2
[43] Syarif I, Prugel-Bennett A, Wills G. SVM parameter optimization using grid search and genetic algorithm to improve classification performance. TELKOMNIKA 2016; 14 (4): 1502. https://doi.org/10.12928/telkomnika.v14i4.3956
[44] Zeng N, Qiu H, Wang Z, Liu W, Zhang H et al. A new switching-delayed-PSO-based opti- mized SVM algorithm for diagnosis of Alzheimer’s disease. Neurocomputing 2018; 320: 195-202. https://doi.org/10.1016/J.NEUCOM.2018.09.001
[45] Tao Y, Yan H, Gao H, Sun Y, Li G. Application of SVR optimized by modified simulated annealing (MSA- SVR) air conditioning load prediction model. Journal of Industrial Information Integration 2018; 15: 247-251. https://doi.org/10.1016/J.JII.2018.04.003
[46] Liu Y, Wang R. Study on network traﬀic forecast model of SVR optimized by GAFSA. Chaos, Solitons & Fractals 2016; 89: 153-159. https://doi.org/10.1016/J.CHAOS.2015.10.019
[47] Zhao W, Tao T, Zio E. System reliability prediction by support vector regression with analytic selection and genetic algorithm parameters selection. Applied Soft Computing 2015; 30: 792-802. https://doi.org/10.1016/J.ASOC.2015.02.026
[48] Kalita DJ, Singh VP, Kumar V. A lightweight knowledge-based PSO for SVM hyper-parameters tuning in a dynamic environment. Journal of Supercomputing 2023; 79 (16): 18777-18799. https://doi.org/10.1007/s11227-023-05385-y
[49] Açıkkar M, Altunkol Y. A novel hybrid PSO- and GS-based hyperparameter optimization algorithm for support vector regression. Neural Computing and Applications 2023; 35 (27): 19961-19977. https://doi.org/10.1007/s00521- 023-08805-5
[50] Wang X, Zhang F, Kung H, Johnson VC, Latif A. Extracting soil salinization information with a fractional-order filtering algorithm and grid-search support vector machine (GS-SVM) model. International Journal of Remote Sensing 2020; 41 (3): 953-973. https://doi.org/10.1080/01431161.2019.1654142
[51] Tang F, Wu Y, Zhou Y. Hybridizing grid search and support vector regression to predict the compressive strength of fly ash concrete. Advances in Civil Engineering 2022; 2022: 3601914. https://doi.org/10.1155/2022/3601914
[52] Abbaszadeh M, Soltani-Mohammadi S, Ahmed AN. Optimization of support vector machine parameters in modeling of Iju deposit mineralization and alteration zones using particle swarm optimization algorithm and grid search method. Computers & Geosciences 2022; 165: 105140. https://doi.org/10.1016/j.cageo.2022.105140
[53] Chi Y, Zhang Y, Li G, Yuan Y. Prediction method of Beijing electric-energy substitution potential based on a grid-search support vector machine. Energies 2022; 15 (11): 3897. https://doi.org/10.3390/en15113897
[54] Mao Y, Wang T, Duan M, Men H. Multi-objective optimization of semi-submersible platforms based on a support vector machine with grid search optimized mixed kernels surrogate model. Ocean Engineering 2022; 260: 112077. https://doi.org/j.oceaneng.2022.112077
[55] Beltrami M, Silva ACL. A grid-quadtree model selection method for support vector machines. Expert Systems with Applications 2020; 146: 113172. https://doi.org/10.1016/j.eswa.2019.113172
[56] Bao Y, Liu Z. A fast grid search method in support vector regression forecasting time series. In: Intelligent Data Engineering and Automated Learning - IDEAL 2006: 7th International Conference; Burgos, Spain; 2006. pp. 504- 511. https://doi.org/10.1007/11875581_61
[57] Yeh IC. Concrete Slump Test. UCI Machine Learning Repository. Irvine, CA, USA: UCI, 2009. https://doi.org/10.24432/C5FG7D
[58] Tsanas A, Xifara A. Energy Eﬀiciency. UCI Machine Learning Repository. Irvine, CA, USA: UCI, 2012. https://doi.org/10.24432/C51307
[59] Yeh IC. Concrete Compressive Strength. UCI Machine Learning Repository. Irvine, CA, USA: UCI, 2007. https://doi.org/10.24432/C5PK67
[60] Brooks T, Pope D, Marcolini M. Airfoil Self-Noise. UCI Machine Learning Repository. Irvine, CA, USA: UCI, 2014. https://doi.org/10.24432/C5VW2C
[61] Kennedy J, Eberhart R. Particle swarm optimization. In: Proceedings of ICNN’95 - International Conference on Neural Networks; Perth, Australia; 1995. pp. 1942-1948. https://doi.org/10.1109/ICNN.1995.488968
[62] Clerc M, Kennedy J. The particle swarm - explosion, stability, and convergence in a multidimensional complex space. IEEE Transactions on Evolutionary Computation 2002; 6 (1): 58-73. https://doi.org/10.1109/4235.985692
[63] Bansal JC, Singh PK, Saraswat M, Verma A, Jadon SS et al. Inertia weight strategies in particle swarm optimization. In: Third World Congress on Nature and Biologically Inspired Computing; Salamanca, Spain; 2011. pp. 633-640. https://doi.org/10.1109/NaBIC.2011.6089659
[64] Hossain SMZ, Sultana N, Irfan MF, Haque SM, Nasr N et al. Artificial intelligence-based super learner approach for prediction and optimization of biodiesel synthesis - A case of waste utilization. International Journal of Energy Research 2022; 46 (14): 20519-20534. https://doi.org/10.1002/er.7764

APA	Açıkkar M (2024). Fast grid search: A grid search-inspired algorithm for optimizing hyperparameters of support vector regression. , 68 - 92. 10.55730/1300-0632.4056
Chicago	Açıkkar Mustafa Fast grid search: A grid search-inspired algorithm for optimizing hyperparameters of support vector regression. (2024): 68 - 92. 10.55730/1300-0632.4056
MLA	Açıkkar Mustafa Fast grid search: A grid search-inspired algorithm for optimizing hyperparameters of support vector regression. , 2024, ss.68 - 92. 10.55730/1300-0632.4056
AMA	Açıkkar M Fast grid search: A grid search-inspired algorithm for optimizing hyperparameters of support vector regression. . 2024; 68 - 92. 10.55730/1300-0632.4056
Vancouver	Açıkkar M Fast grid search: A grid search-inspired algorithm for optimizing hyperparameters of support vector regression. . 2024; 68 - 92. 10.55730/1300-0632.4056
IEEE	Açıkkar M "Fast grid search: A grid search-inspired algorithm for optimizing hyperparameters of support vector regression." , ss.68 - 92, 2024. 10.55730/1300-0632.4056
ISNAD	Açıkkar, Mustafa. "Fast grid search: A grid search-inspired algorithm for optimizing hyperparameters of support vector regression". (2024), 68-92. https://doi.org/10.55730/1300-0632.4056

APA	Açıkkar M (2024). Fast grid search: A grid search-inspired algorithm for optimizing hyperparameters of support vector regression. Turkish Journal of Electrical Engineering and Computer Sciences, 32(1), 68 - 92. 10.55730/1300-0632.4056
Chicago	Açıkkar Mustafa Fast grid search: A grid search-inspired algorithm for optimizing hyperparameters of support vector regression. Turkish Journal of Electrical Engineering and Computer Sciences 32, no.1 (2024): 68 - 92. 10.55730/1300-0632.4056
MLA	Açıkkar Mustafa Fast grid search: A grid search-inspired algorithm for optimizing hyperparameters of support vector regression. Turkish Journal of Electrical Engineering and Computer Sciences, vol.32, no.1, 2024, ss.68 - 92. 10.55730/1300-0632.4056
AMA	Açıkkar M Fast grid search: A grid search-inspired algorithm for optimizing hyperparameters of support vector regression. Turkish Journal of Electrical Engineering and Computer Sciences. 2024; 32(1): 68 - 92. 10.55730/1300-0632.4056
Vancouver	Açıkkar M Fast grid search: A grid search-inspired algorithm for optimizing hyperparameters of support vector regression. Turkish Journal of Electrical Engineering and Computer Sciences. 2024; 32(1): 68 - 92. 10.55730/1300-0632.4056
IEEE	Açıkkar M "Fast grid search: A grid search-inspired algorithm for optimizing hyperparameters of support vector regression." Turkish Journal of Electrical Engineering and Computer Sciences, 32, ss.68 - 92, 2024. 10.55730/1300-0632.4056
ISNAD	Açıkkar, Mustafa. "Fast grid search: A grid search-inspired algorithm for optimizing hyperparameters of support vector regression". Turkish Journal of Electrical Engineering and Computer Sciences 32/1 (2024), 68-92. https://doi.org/10.55730/1300-0632.4056