Both results are correct. As Luboš Motl pointed out in his comment, we can get the geometrical optics approach answer from the wave method answer by averaging it over 1 full period.
Perhaps you made a mistake somewhere when averaging.
If we calculate the average carefully:
$\bar{T}=\frac{1}{2\pi}\int _{-\pi}^{\pi}T d\varphi$
$\bar{T}=\frac{1}{2\pi} \times T_{01}T_{12} \int_{-\pi}^{\pi}\frac{1}{(r_{10}r_{12})^2-2r_{10}r_{12}cos\varphi+1}d\varphi$
we'll get this
$\bar{T}=\frac{1}{2\pi} \times T_{01}T_{12}\left|\frac{2tan^{-1}\left(\frac{1+r_{10}r_{12}}{1-r_{10}r_{12}}tan{\frac{\varphi}{2}}\right)}{1-(r_{10}r_{12})^2}\right|_{-\pi}^{\pi} $
you can check here
now it's not hard to see that
$\bar{T}=\frac{1}{2\pi} \times T_{01}T_{12}\frac{(2\pi)}{1-R_{10}R_{12}}=\frac{T_{01}T_{12}}{1-R_{10}R_{12}}$
same as the one obtained from geometric optics method
EDIT(to respond to the EDIT part of the question):
First of all, let's make everything clear so that the equation that you quoted from Wikipedia make sense. The amplitude here means the magnitude of electric field. The $t$ simply means the ratio of transmitted and incoming electric field (We were using different definition of $t$ in the calculations above, the $t$ that we use above includes other factors beside the ratio of electric fields). And the transmittance $T$ here represents the fraction of power transmitted to the medium 2. Here the power $P$ is proportional to
$P\propto IA$
where the intensity is proportional to $I\propto nE^2$. And the beam area is proportional to $A\propto cos\theta$, it's because the beam cross section gets smaller as it bends toward the boundary plane (see this image). Putting them together we have
$P\propto nE^2cos\theta$
so from the new definition of transmittance $T$ we get
$T=\frac{n_2 cos\theta_t}{n_1 cos\theta_i}t^2$
and since the incoming and reflected rays have the same $cos\theta$ and $n$, the reflectance is simple
$R=r^2$
we have calculated that
$\bar{t^2}=\frac{t_{01}^2t_{12}^2 }{1-R_{10}R_{12}}$
multiply both sides with $\frac{n_2 cos\theta_t}{n_0 cos\theta_i}$
$\bar{T}=\frac{t_{01}^2t_{12}^2 }{1-R_{10}R_{12}}\frac{n_2 cos\theta_t}{n_0 cos\theta_i}$
We can check
$T_{01}T_{12}=\left(\frac{n_1 cos\theta_m}{n_0 cos\theta_i}t_{01}^2\right)\left(\frac{n_2 cos\theta_t}{n_1 cos\theta_m}t_{12}^2\right)=\frac{n_2 cos\theta_t}{n_0cos\theta_i}t_{01}^2t_{12}^2$
Thus again we have
$\bar{T}=\frac{T_{01}T_{12}}{1-R_{10}R_{12}}$
The rays will only come exactly to the focus if you're using a parabolic mirror. A spherical mirror is considered a good compromise for limited fields of view (plus it's got the advantage of behaving the same off-axis as on-axis).
Your second setup is invalid, I believe. You need to differentiate between the object's position (or height) and the height of both rays as they reach the mirror. I may have misunderstood your diagram, in which case I apologize.
Best Answer
In Wikipedia the caustic is defined as follows.
In optics, a caustic is the envelope of light rays reflected or refracted by a curved surface or object, or the projection of that envelope of rays on another surface.
You can think of the envelope of a family of curves as a curve that is a tangent to each of them.
Here is a diagram on page 60 of "A Treatise on Optics" by David Brewster published in 1831.
There is a source of light at $R$ and the rays are reflected a circular surface $MN$ although only the rays reflected from part $MB$ are shown.
The angle of incidence is equal to the angle of reflection for each of the rays.
None of the rays meet the principal axis at the same point although those that hit the reflecting surface at positions $B, 1, 2$ and $3$ do so approximately and this is where the image of the object $R$ is formed.
The concave mirror is suffering from a mirror defect called spherical aberration.
Now if you look at all the rays in the upper half they define an area with no light and an area where there is light and in particular there is an envelope, shown in red at the bot one in the top half, which defines places where the rays are at a tangent - this is the caustic curve and here is a photograph of one although you may well see one very frequently in you cup of coffee or tea?
The caustic curve is particularly noticeable because it is brighter because there are more rays passing through that region then in other regions around the reflector.
In the diagram above one half of the imaginary caustic I have coloured blue.
This is obtained by moving the object to the other side of the reflector $MBN$ so that it is acting like a convex mirror. The reflected rays diverge but when back produced form the blue imaginary caustic $Nf'M$.
In recent years caustics have been used to find exoplanets.
A star and its orbiting plant act as a gravitational lens and form a caustic in a plane at right angled to the plane of the Earth's orbit around the Sun which is shown in blue in the left-hand diagram.
As the orbit of the Earth (shown in red) crosses the caustic the light intensity resulting from the caustic changes as shown in the graph of intensity against time.
Although the shape of the caustic is very complex that very complexity means that from the caustic information about the star and its exoplanet can be obtained.
The edX course "ASTRO2x Astrophysics: Exploring Exoplanets" is a good introduction to this effect and many others which are used to discover exoplanets and their properties.