Is the camera plane the projective space of the real world?
It depends.
Usually a physical camera has a limited sensor, so the thing where the image appears is too bounded to be an affine or projective plane. You can say “well, simply extend that plane infinitely”. Then it depends on how exactly you define that extension, whether you end up with an affine or a projective space. If you say “well, take any line on it, and for any distance $d\in\mathbb R$ take the point at position $d$ along that line to be a point of my plane as well” then the result is affine. If you say “well, take any pair of lines on it, and include their point of intersection as well” then it's projective.
And for something planar to be “the projective space” of “the real world”, that real world has to be planar, too. So if you have a camera image of things happening on the flat surface spanned by your table top, then yes, you could obtain an image of that surface in your projective camera plane. But if you take a camera image of a 3d scene, you get a projection, which is a reduction of information and usually not what projective geometry describes. Usually, a projective space is used as something more complete than the underlying affine space: you add points (at infinity), you don't remove them (by combining multiple points in the line of projection into a single point on the image plane).
Is the line which is the image of the horizon the distinguished line?
If you have your geometry sketched on a real paper lying on your real (horizontal) table, then the horizon of your real world is the distinguished line. You draw two parallel lines on your table, you look from an angle and you can almost see them converging on the horizon. Of course, that horizon is infinitely far away, so if you were to go there, the fact that the earth is not flat may break the simile.
If you take a camera image of that situation, then your projective camera sensor plane has its own distinguished line at infinity, namely where two parallel lines in the camera sensor plane intersect. If the image of one distinguished line is the other distinguished line, then your camera plane is parallel to the real world table plane, and the resulting transformation would be called affine. Otherwise it's a generic projective transformation.
Whenever we do an Affine transform do we need to look out for a distinguished line?
It depends on how and why you do affine transformations.
If you do an affine transformation via a multiplication of a matrix of the form $$\begin{pmatrix}*&*&*\\*&*&*\\0&0&1\end{pmatrix}$$ then the form of the matrix already guarantees that points with a $z$ coordinate of zero will get mapped to points which again have a $z$ coordinate of zero, i.e. using the $z=0$ line as the distinguished line in both your spaces is already implied by the form of the transformation.
If you do affine transformations as $$\begin{pmatrix}x\\y\end{pmatrix}\mapsto\begin{pmatrix}ax+by+c\\dx+ey+f\end{pmatrix}$$ then you are using the same scheme computation as above, plus you are using non-homogeneous coordinates, so you might not even see what this $z$ coordinate is about since the 2d affine coordinates don't even allow you to express points at infinity.
If, on the other hand, you have some weird choice of coordinate systems where the distinguished line has a more complicated form, and that form perhaps isn't the same for source and target space of your transformation, and you need to ensure that your transformations are indeed affine, then yes, you need to be explicit about the lines at infinity, and verify that they get mapped correctly. One common example would be trilinear coordinates, where $x+y+z=0$ represents the line at infinity. Mapping from carthesian to trilinear coordinates and back is a situation where you want to make sure that the line at infinity gets transformed correctly.
Why does just a distinction of the geometry a line in the perspective plane make the geometry an Affine geometry?
As the other answers by Joseph Malkevitch and by AdiPiratla have already indicated, there are different ways to treat the relation between affine and projective geometry. One is to say that you get projective geometry from affine geometry if you add a point at infinity for every bundle of parallel lines, and a line at infinity made up from all these points. In this sense, a projective space is an affine space with added points.
Reversing that process, you get an affine geometry from a projective geometry by removing one line, and all the points on it. By convention, one uses the line $z=0$ for this, but it doesn't really matter: the projective space does not depend on the choice of coordinates, and removing any line will turn it into an affine space.
But often one wants to exploit the machinery from projective geometry to perform affine operations. Perhaps you want to combine affine and projective transformations, or some such. In such a setup, you can say that as long as you keep track of which line is the line at infinity, you know how to get from there to affine geometry, so you are already doing affine geometry in a different representation. Affine geometry is like projective geometry with one line (the “distinguished line”) labeled “remove this to obtain an affine plane”. In this sense, an affine space is a projective space with additional information.
First, "ray through the origin" is (imo) a bad term to use. Indeed a point in $\mathbb{P}^2$ corresponds to a set of the form $\{k (x_1, x_2, x_3) : k \in \mathbb{R}\}$ for some nonzero vector $(x_1, x_2, x_3)$, but this is a line, not a ray! The distinction is very important, since $(x_1, x_2, x_3)$ and $(-x_1, -x_2, -x_3)$ trace out the same line (and thus correspond to the same point in $\mathbb{P}^2$).
Anyway, this is really a way to define "line in $\mathbb{P}^2$". A line in $\mathbb{P}^2$, by this definition, is the set of points in $\mathbb{P}^2$ corresponding to lines through the origin in $\mathbb{R}^3$ which are contained in a given plane through the origin.
Best Answer
The sentence that begins, “However, in projective space, there is no concept...” is the key. You might be used to constructing, say, the projective plane $\mathbb{RP}^2$ in a “bottom-up” manner by starting with the Euclidean plane $\mathbb R^2$ and then adding “points at infinity” and the line that they all lie on. Hartley and Zisserman are instead taking a “top-down” approach here: starting with the projective plane, complete with all of those “extra” points, as a given. They point that they’re trying to make is that from this point of view there’s nothing special about those particular points—we can pick any line in the projective plane to be our “line at infinity” with a different set of lines that are then considered to be parallel for each choice.
A different model of the projective plane might help here. Instead of starting with the Euclidean plane and adding points, start with the unit sphere in $\mathbb R^3$ and identify antipodes. (This is equivalent to the model in which lines in $\mathbb R^3\setminus\{0\}$ are considered to be points in $\mathbb{RP}^2$.) Lines in the projective plane then correspond to great circles on this sphere, but with this construction there’s nothing special about any particular great circle. The definition of parallelism depends on knowing which of these lines is the “line at infinity,” so until you’ve chosen a particular line to be that, it doesn’t make any sense to talk about lines being parallel. Making that choice imposes an affine geometry on the space—you now know when lines are parallel—as the authors discuss in more detail later.
You can see this in action when you apply a projective transformation to the Euclidean plane. If the transformation maps the line at infinity to some other line, then parallel lines in the source don’t generally remain parallel in the image: they instead converge at a finite point (their “vanishing point”) that lies on the image of the line at infinity. (Identifying this image will become important in later chapters.) Parallelism is not a projective-geometric propery—it is not preserved by projective transformations. In a sense, considering points with homogeneous coordinates of the form $x:y:0$ as the points at infinity is an artifact of the coordinate system that you’ve chosen; the projective map in this paragraph can be viewed passively as a change of basis rather than actively as a “warping” of the plane. We could just as well have said that points with homogeneous coordinates of the form $0:y:w$ are at infinity instead, and some sources do just that.