Please bear in mind that I am an experimentalist, therefore I treat theoretical ideas and models as dependent on experimental observations and not vice versa.
Facts:
elementary particles have measurable quantum numbers. These quantum numbers define the particle.
elementary particles have mass
there exist elementary and composite (from elementary) particles that have the same mass as others but opposite charge, as the electron vs positron, proton vs antiproton, called antiparticles because:
there is a very high probability that when scattered against each other, the particle and antiparticle disappear and the energy appears in a plethora of other particles. This is called annihilation because charge disappears, and in general the opposite quantum numbers are "annihilated".
On the theoretical side these experimental observations are fitted beautifully by the Dirac equation if one considers the negative energy solutions to describe the antiparticles.
One thing we can be sure of is that measured antiparticles travel forward in time.
The popular theoretical interpretation of antiparticles being particles traveling backwards in time mainly comes from the Feynman diagrams. These are a brilliant mathematical tool for fitting and predicting measurements representing particle interactions as incoming lines and outgoing lines and in between lines representing virtual particles that carry the quantum numbers but are off mass shell.
Due to the CPT theorem, once a Feynman diagram is drawn, one can interpret the lines consistently according to CPT and will get the corresponding cross sections and probabilities for the change in the quantum numbers consistent with CPT (charge conjugation, parity and time reversal). Identifying a positron as a backwards-in-time electron is an elegant interpretation that in the Feynman diagrams exhibits the CPT symmetry they must obey.
What I am saying is, the statement "positrons are backward-going electrons" is a convenient and accurate mathematical representation for calculation purposes. "As if". There has not been any indication, not even a tiny one, that in nature (as we study it experimentally) anything goes backwards in time, as we define time in the laboratory.
Edit replying to comment by Nathaniel:
I'm curious: how would you expect empirical data from a backwards-travelling positron to differ from what we actually see?
In this bubble chamber picture we see the opposite to annihilation, the creation by a photon of an e+ e- pair. (This is an enlarged detail from the bubble chamber photo in the archives. The original web page with the letterings has disappeared, as of july2017)
The magnetic field that makes them go into helices is perpendicular to the plane of the photo. We identify the electron by the sign of its curvature as it leaves the vertex. The positron is the one going up to the left corner. We know it is not an electron that started its life before the vertex formed because as an electron/positron moves through the liquid it loses energy and the loss defines the time direction of the path. So the particle has to start at the vertex and end at the upper left, so it has the opposite curvature to the electron and it is a positron.
A Feynman diagram looks like a scattering in real space, or a pair production, but one cannot project the intricacies of the mathematics it represents onto real space. It is only the calculations of cross sections and probabilities that can be compared with measurements.
The reasoning in the question is correct. If you have a box with gas particles placed in half of a box but otherwise uniformly random and with random velocities then it is overwhelmingly likely that it entropy will increase with time, but if reverse the velocities, you will still have randomly distributed velocities and the same argument will apply. By time symmetry reversing the velocities and going forward in time is equivalent to going backward in time. So system prepared as described above would almost certainly be in local entropy minimum wrt to time.
If the whole universe only consisted of some water with unevenly distributed dye in it, and we knew nothing about its origin, then inferring that the dye was more evenly distributed in the past would be rational. The water and dye being in a beaker near a teacher in a far from equilibrium universe makes other explanations much more likely though. However, your line of reasoning has some bite at the cosmological level. This is the Boltzmann Brain Problem. It is still not satisfactorily resolved, as you can see on ArXiv.
The second law of thermodynamics works (and is a law) because the universe is far from equilibrium (ie low entropy) and is believed to have started much farther from equilibrium that than it is now. Of course a big part of the reason for believing that is the second law. ;)
Here is a more detailed explanation from my answer to Where does deleted information go?:
The apparent conflict between macroscopic irreversibility and microscopic reversibilty is known as Loschmidt's paradox, though it is not actually a paradox.
In my understanding sensitivity to initial conditions, the butterfly effect, reconciles macroscopic irreversibility with microscopic reversibility. Suppose time reverses while you are scrambling an egg. The egg should then just unscramble like in a film running backwards. However, the slightest perturbation, say by hitting a single molecule with a photon, will start a chain reaction as that molecule will collide with different molecules than it otherwise would have. Those will in turn have different interactions then they otherwise would have and so on. The trajectory of the perturbed system will diverge exponentially from the original time reversed trajectory. At the macroscopic level the unscrambing will initially continue, but a region of rescrambling will start to grow from where the photon struck and swallow the whole system leaving a completely scrambled egg.
This shows that time reversed states of non-equilibrium systems are statistically very special, their trajectories are extremely unstable and impossible to prepare in practice. The slightest perturbation of a time reversed non-equilibrium system causes the second law of thermodynamics to kick back in.
The above thought experiment also illustrates the Boltzmann brain paradox in that it makes it seem that a partially scrambled egg is more likely to arise form the spontaneous unscrambling of a completely scrambled egg than by breaking an intact one, since if trajectories leading to an intact egg in the future are extremely unstable, then by reversibility, so must trajectories originating from one in the past. Therefore the vast majority of possible past histories leading to a partially scrambled state must do so via spontaneous unscrambling. This problem is not yet satisfactorily resolved, particularly its cosmological implications, as can be seen by searching Arxiv and Google Scholar.
Nothing in this depends on any non classical effects.
Best Answer
Antimatter increase in entropy over time. We can verify this with a thought experiment. Take ten positrons. Put five in one side of a chamber with a barrier and then the other 5 on the other side of the barrier in the same chamber. The chamber and barrier are also made of antimatter. The positrons repel each other and so each have a certain amount of kinetic energy due to changes in their potential energy. Now, remove the barrier and see if they tend to randomly assort themselves throughout the chamber. Because they are more likely to be found in disordered arrangements over time we know that the entropy of this system is increasing with time.
What we wouldn't observe, is all of the positrons moving back to where they started and not mixing at all. Not that it's impossible, just extremely unlikely.