In both your "faulty code" and your "detour" you are using matlab's 'sqrt' function to obtain 'H12', and apparently in each case the answer is not always appropriate for your computations with it. The rule followed by 'sqrt' for complex numbers is that the real part of the square root is never negative. Another way of expressing that is that the 'angle' of the square root result always falls in the interval -pi/2 < angle <= +pi/2.
Consequently in searching for a "pattern" of what your procedure considers as incorrect, you should probably be investigating the 'angle' of H12 that would be correct versus incorrect. For example, suppose it should always satisfy 0 <= angle < pi. In that particular case, the necessary correction would be this:
H12 = sqrt(Z1.*Z2);
H12 = abs(H12).*exp(1i*mod(angle(H12,pi));
We who are responding to your question cannot of course tell what values would be appropriate, but you are in a position to do so. Just make a record of the angles of H12 for those points on your plot that produce the wrong result, and devise an appropriate correction by pi in the angle of H12.
Best Answer