It all boils down to how much information there is to, in principle, distinguish which way the photon went from the final state of the beam splitter, as encoded in the overlap between its two possible final states. The interference is destroyed because the photon gets entangled with the beam splitter, and the amount of entanglement depends on this overlap.

Say, then that if the photon goes straight through the beam splitter, to state $|{\to}\rangle$, the beam splitter stays put, at state $|0\rangle$, whereas if the photon gets deflected into state $|{\downarrow}\rangle$, the beam splitter gets some upwards momentum, $|{\Uparrow}\rangle$. If the result is a superposition, then, the total state of the system is entangled:
$$|\Psi\rangle=|\to\rangle|0\rangle + |{\downarrow}\rangle|{\Uparrow}\rangle .$$

Regardless of what you do to the beam splitter - i.e. measure its state or just forget about it - in the absence of a measurement that introduces further interactions, the information you have available to produce an interference pattern on the photon side is given by the reduced density matrix obtained by taking the partial trace over the beam splitter.

Calculating this object is fairly simple. In the $\{|{\to}\rangle, |{\downarrow}\rangle\}$ basis, it is given by
$$
\rm{Tr}_{\rm{BS}}(|\Psi\rangle\langle\Psi|)
=
\begin{pmatrix}
1 & \langle0|{\Uparrow}⟩ \\ \langle{\Uparrow}|0⟩&1
\end{pmatrix}.
$$
If the beam splitter states are completely distinguishable, then they are orthogonal and what you get on the photon side is a completely mixed state, $|{\to}⟩⟨{\to}|+|{\downarrow}⟩⟨{\downarrow}|$, which is completely classical, and from which no interference can be extracted. Note that this happens regardless of whether you actually measure the beam splitter's momentum or not.

If there is no effect on the beam splitter, on the other hand, the states are the same, and the photon's density matrix corresponds to a pure state, $\left(|{\to}⟩+|{\downarrow}⟩\right)\left(⟨{\to}|+⟨{\downarrow}|\right)$. Then you will see complete interference, but you will have no "which way" information available, even in principle.

In any physical realization, of course, you're somewhere in the middle. Most realizations have very similar states for the beam splitters, which means that $\langle{\Uparrow}|0\rangle$ is very close to 1, and you get good interference, but as the states become more distinguishable, the contrast in the interference fringes is reduced.

I understand this can feel pretty thin. After all, how are we to know that we've eliminated all possible places where "which way" information may *in principle* be available? This is in fact how it goes down in the lab, and that's the reason observing things like Mandel dips is very, very touchy: if you want two photons to interfere, you need to make sure that they truly are indistinguishable - in spatial profile, displacement, spectrum, and timing - for otherwise there will be (possibly undetected) entanglement with some other mode, and that will reduce or destroy your interference contrast.

I assume that you're building an infinity corrected microscope (because I don't see why else you would need a beam splitter)?

If that is the case, I would immediately assume you're using a cube beam splitter? If so, replace it with a plate beam splitter, which would eliminate the ghosts, because there would be no optical surfaces perpendicular to the optical axis. Take into consideration that a plate will displace your optical axis laterally, so you'll have to compensate for that.

## Best Answer

Nobody is answering this question, so I'll take a stab at it.

Consider the mirror. Suppose you started your experiment by (somehow) putting it in a nearly-exact momentum state, meaning there is a large uncertainty in its position. Now, when you send a photon at it, the photon either bounces off or passes through. If the photon bounces off the mirror, it will change the momentum of the mirror. You could theoretically measure the "which-way" information by measuring the momentum of the mirror after you've done the experiment. In this scenario, there wouldn't be any interference.

However, you didn't do that. You started the mirror off in a thermal state at room temperature. This state can be considered as a superposition of different momentum states of the mirror

^{1}, with a phase associated to each one. If you change the momentum by a small amount, the phase associated to this state in the superposition only changes by a small amount. Now, let $p_\gamma$ and $p_m$ be the original momenta of the photon and the mirror, and let $\Delta p_\gamma$ be the change in the momentum when the photon bounces off the mirror. When you send the photon towards the mirror, the original state $p_m$ (photon passes through) will end up in the same configuration as the original state $p'_m = p_m - \Delta p_\gamma$ (photon bounces off). These two states $p_m$ and $p'_m$ had nearly the same phase before you aimed the photon at the mirror, so they will interfere, and if the phase on these two states are really close, the interference will be nearly perfect.Of course, a change in momentum isn't the only way for the mirror to gain which-way information. However, I think what happens when you consider the other ways is that they behave much like this, only not anywhere near as cleanly, so they're harder to work with.

^{1}Technically, it's a mixed state, i.e., a density matrix, and not a pure state. But the basic idea of the above explanation still holds.