Description
Classifiers that find the widest-margin boundary between classes, allowing tuned violations and kernel-based non-linear boundaries, evaluated by held-out ROC curves.
Support vector machines build on the maximal margin classifier, which chooses the separating hyperplane with the widest margin to the nearest training points (the support vectors). Because real classes overlap, the support vector classifier allows slack within a total violation budget C: a large budget gives a wide margin, many support vectors, low variance and high bias. Replacing inner products with a kernel (polynomial of degree d, or radial with parameter gamma) yields non-linear boundaries without computing an explicit feature space. C and kernel parameters must be tuned by cross-validation; training ROC curves flatter flexible kernels, so comparisons belong on held-out data. Multi-class problems use one-versus-one or one-versus-all schemes. Results are often close to logistic regression, which is preferred when classes overlap heavily and probabilities are needed.
Attributes
- Discipline
- Computational & Machine Learning
- Loss type
- Siloed
- Stage
- Discovery, Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- Code the two classes as +1 and -1 and standardise features.
- Choose a kernel: linear, polynomial of degree d, or radial with parameter gamma.
- Tune C (and d or gamma) by k-fold cross-validation over a grid.
- Fit on training data; note that the boundary depends only on the support vectors.
- Classify new cases by the sign of the fitted function and use fitted values as scores for ROC and AUC on held-out data.
- For more than two classes use one-versus-one voting or one-versus-all largest score, and compare against logistic regression.
Sources
- An Introduction to Statistical Learning with Applications in Python (James et al., 2023)
- Pattern Classification (Duda et al., 2001)
- Pattern Recognition and Machine Learning (Bishop, 2006)
- Computer Vision: A Modern Approach (Forsyth & Ponce, 2011)
Also known as
- SVM
- Support vector classifier
- Soft-margin classifier
- Maximal margin classifier
- Kernel methods
- Support vector regression
Categories
- Analytics & Behaviour
- Computational & Machine Learning