To introduce classifiers we will assume that ~x is a vector of real-valued numerical input features. There are p of them. The variable we are trying to learn is called the response, Y . Inititially we will assume that Y is a binary class and it will simplify the book-keeping to say that the two classes are +1 and −1. In linear classification, we seek to divide the two classes by a linear separator in the feature space. If p = 2 the separator is a line, if p = 3 it is a plane, and in general it is a p − 1 dimensional hyperplane in a p dimensional space;
We can specify the orientation of a plane by a vector w~ in the direction perpendicular to it. There are infinitely many parallel planes with the same orientation. We need to pick out one of them, which we can do by fixing its distance from the origin, b say.
R code to generate data separated by:
x1 &lt;- runif(30,-1,1) x2 &lt;- runif(30,-1,1) x &lt;- cbind(x1,x2) Y &lt;- ifelse(x2&gt;0.5+x1,+1,-1) plot(x,pch=ifelse(Y&gt;0,"+","-"), + xlim=c(-1,1),ylim=c(-1,1),cex=2) abline(0.5,1) &gt; points(c(0,0),c(0,0),pch=19) lines(c(0,-0.25),c(0,0.25),lty=2) arrows(-0.3,0.2,-0.4,0.3) text(-0.45,0.35,"w",cex=2) text(-0.0,0.15,"b",cex=2)
Data with linear separator
Figure: Example of a linear separator: (+ and -) indicate points in the positive and negative classes.
The perpendicular vector w~ fixes the orientation of the separator. The directed distance b from the origin fixes the position of the separator.
Using a separator to predict class
Given any separator, ( , b), we would like a formula to express the idea that points on one side of the separator are predicted to belong to one class whilst those on the other side are predicted to belong to a different class. We can do this via: Yˆ (~x, b, w~ ) = sign(b + ~x · w~ ) where, as a reminder, ~x · w~ = Pp i=1 xiwi , is the dot product and sign is a function which returns +1 if its argument is > 0 and −1 if its argument is < 0. Remember that if w~ is a unit vector, then ~x · w~ is the projection of ~x on to w~ , so this is in fact checking whether ~x is on the positive side of the separating plane or not.