I am reading Bishop - Pattern Recognition and Machine Learning book. In Chapter 4 first page, there is a statement that
In this chapter, we consider linear models for classification, by which we mean that the decision surfaces are linear functions of the input vector x and hence are defined by (D − 1)-dimensional hyper planes within the D-dimensional input space.
If the input vector is of D dimensions then the decision boundary should also be of D dimensions. Why is it D-1? For example, in 2-D plane, the line separating the points is also a straight line which is of 2 dimensions.
Am I thinking something terribly wrong? Can anyone explain me how is it D-1 dimensions for decision boundary?
Best Answer
The line is a 1-D boundary in 2-D space. If you think of yourself as a point on the decision boundary, the number of (non-parallel nor anti-parallel) directions you could travel on the boundary will be its dimensions. With a line you can go forward or backward (which is anti-parallel to forward), so there is only one dimension. A 2-D plane would let you go forward-backwards or left-right which makes it 2-D.
In the images below the decision boundaries for 2D and 3D input space are shown in blue. The orange lines illustrate that these decision boundaries have D$-1$ dimensions themselves.
Think about rotating the image. You can rotate it so that the blue line becomes horizontal (i.e. super clearly 1D) but no matter how you rotate it the plane created by the black axes will still be 2D as seen in this image. This is because the blue line is a 1D slice in 2D space: