There is some freedom in deciding which particle in a particle-antiparticle pair is called "matter" and which is called "antimatter" but the freedom is smaller than you think. A basic problem is that your sentence
Antimatter of course annihilates ordinary matter, but the more precise statement is that antiparticles annihilate the same types of particles.
isn't really true. In fact, the opposite statement, while inaccurate, is much closer to the truth: matter and antimatter often can annihilate even if they belong to different species.
For example, a proton will rapidly annihilate with an antineutron (or an up-quark with anti-down-quark, if we look at the same process at the quark level), leaving some positron and neutrino (whose rest mass is much lower than the rest mass of either proton or antineutron) with lots of energy.
The up-quark and down-quark are different species or flavors but it would make no sense to call one of them "matter" and the other "antimatter" because they can "almost annihilate" to "almost nothing". More generally, all quark flavors are similar and it's better to call all of them "matter", especially because they may be related by symmetries that don't have a reason to include charge conjugation C.
Now, the atoms are composed of protons and electrons – and we call the atomic bound states "matter". That implies that an electron has the same "pro-anti" label as the six quarks. There are no bound states between positrons and protons so there would be no atomic "matter" if you flipped the convention for electrons but not protons.
Grand unified theories actually do link some 2-component spinors to larger representations and these representation contain fields that create both matter and antimatter so the binary label "pro-anti" becomes more subtle in such theories. We must still carefully distinguish a field and its Hermitian conjugate.
The "pro-anti" dichotomy remains meaningless for some particles, anyway. There are totally neutral particles – photons, Z-bosons, gluons, gravitons – that are identical to their antiparticles so here there is no "polarization", of course. There are also charged particles, W-bosons, for which it makes no sense to ask which of them is matter and which of them is antimatter. A W-boson may decay to a quark-antiquark pair so it's equally "far" from matter as it is from antimatter.
Neutrinos seem to be Majorana particles so far so they are identical to their antiparticles, too. However, the helicity (left-handed, right-handed) is correlated with the usual labels "pro-anti" which means that we can distinguish matter from antimatter, after all. There can also be right-handed neutrinos in which case the separation of neutrinos to "matter" and "antimatter" is exactly as possible (in principle) as it is for electrons and positrons.
Strictly speaking, it is indeed incorrect that neutrinos travel at "close to the speed of light". As you said, since they have mass they can be treated just like any other massive object, like billiard balls. And as such they are only traveling at nearly the speed of light relative to something. Relative to another co-moving neutrino it would be at rest.
However, the statement is still true for almost all practical purposes. And it doesn't even matter in which reference frame you look at a neutrino. The reason is that a non-relativistic neutrino doesn't interact with anything. Or in other words: all the neutrinos you can detect necessarily have to have relativistic speeds.
Let me elaborate. Since neutrinos only interact weakly they are already extremely hard to detect, even if they have high energies (> GeV). If you go to ever lower energies the interaction cross-section also decreases more and more. But there is another important point. Most neutrino interaction processes have an energy threshold to occur. For example, the inverse beta decay
$$ \bar\nu_e + p^+ \rightarrow n + e^+$$
in which an antineutrino converts a proton into a neutron and a positron, and which is often used as a detection process for neutrinos, has a threshold of 1.8 MeV antineutrino energy. The neutron and the positron are more massive than the antineutrino and the proton, so the antinneutrino must have enough energy to produce the excess mass of the final state (1.8 MeV). Below that energy the (anti)neutrino cannot undergo this reaction any more.
A reaction with a particularly low threshold is the elastic scattering off an electron in an atom. This only requires a threshold energy of the order of eV (which is needed to put the electron into a higher atomic energy level). But a neutrino with eV energies would still be relativistic!
Assuming that a neutrino has a mass of around 0.1 eV, this would still mean a gamma factor of $\gamma\approx 10$. For a neutrino to be non-relativistic it would have to have a kinetic energy in the milli-eV range and below. This is the expected energy range of Cosmic Background Neutrinos, relics from the earliest times of the universe. They are so to say the neutrino version of the Cosmic Microwave Background. So not only do non-relativistic neutrinos exist (according to mainstream cosmological models), they are also all around us. In fact, their density at Earth is $\approx$50 times larger than neutrinos from the Sun!
There is a big debate if they can ever be detected experimentally. There are a few suggestions (and even one prototype experiment), but there are differing opinions about the practical feasibility of such attempts. The only process left for neutrinos at such small energies is neutrino-induced decay of unstable nuclei. If you have an already radioactive isotope, it's like the neutrino would give it a little "push over the edge". The $\beta$-electron released in the induced decay would then receive a slightly larger energy than the Q-value of the spontaneous decay and the experimental signature would be a tiny peak to the right of the normal $\beta$-spectrum. This will still be an extremely rare process and the big problem is to build an apparatus with a good enough energy resolution so that the peak can be distinguished from the spectrum of normal spontaneous nuclear decay (amidst all the background).
The Katrin experiment is trying to measure the endpoint of $\beta$-spectrum of Tritium in order to determine the neutrino mass. But under very favorable circumstances they even have some chance to detect such a signature of cosmic background neutrinos.
TL;DR: In fact there are non-relativistic neutrinos all over the place, but they they interact so tremendously little that they seem to not exist at all.
Best Answer
To maintain lepton number as a conserved quantity.
Consider, in detail, what's going on in a beta decay (well, I'm going to ignore the nuclear context). The reaction is then $$ n \longrightarrow p^+ + e^- + \nu \,,$$ where you should take the symbol $\nu$ to mean some neutrino (without prejudice about matter-type or anti-matter-type for the moment).
There are zero leptons in the LHS of the equation, and on the RHS there is the electron (which counts as one lepton) and the neutrino. If we want lepton number to be conserved in the reaction (we do) then that has to have lepton number -1, which makes it an anti-particle. So, written properly the reaction is $$ n \longrightarrow p^+ + e^- + \bar{\nu} \,.$$
Later still it was recognized as necessary to assign a flavor to the neutrino and the symbol becomes $\bar{\nu}_e$, but that's another story.