Skip to the content.

Deep Learning in Science and Engineering

Rajib Mukherjee, Texas A&M Energy Institute
Optimal Hyperparameter Estimation in Support Vector Machine

Authors

Abstract

Machine learning algorithms are intended to account for loss minimization over training instances. When the data is heteroscedastic (i.e., variance of the output depends on the input), weighted instance learning algorithms are found to be useful (Kersting et al. 2007). Weighted instance learning is also used for variable ranking in order of their relevance (Herbrich et al. 2000). Generally, the weights, also called hyperparameters, are qualitatively selected, where small noise variance are associated with larger weights and vice versa. In these cases, it is difficult to find how large or small the weights should be. Determining optimal hyperparameters is a model selection procedure and can impact the result significantly. In the present work, hyperparameter estimation will be achieved through multi-parametric optimization technique. Multi-parametric programming has been successfully implemented in model predictive control (MPC) (Alberto et al. 2002). But the application in other branches, especially machine learning (ML) is limited. In this work, the application of multiparametric optimization on optimal hyperparameters selection has been verified through the performance of Support vector machine (SVM) in fault classification of chemical process data obtained from Tennessee Eastman Process (TEP). In SVM classification learning, the instance weights as well as margin control parameter are hyperparameters that needs to be optimal in order to minimize misclassification error. The results are demonstrated using \(C-SVM\) (instance weights as hyperparameters) as well as \(v-SVM\) (instance weights and \(v\) for margin control as hyperparameters).