I don't get why you used a product $ab$. It suffices to just consider the two paths where $(h_1, h_2) = (t,t)$ for $t> 0$ and $(h_1, h_2) = (t,-t)$ for $t>0$. Then, show that the limit as $t \to 0^+$ along these two directions yields different answers. So, basically, the idea of your computation is right, but the presentation seems weird.
Anyway, here's a more general overview of why your function fails to be holomorphic.
Let $U \subset \Bbb{C}$ be open and $f:U \to \Bbb{C}$ be a map. Then, $f$ is holomorphic at a point $\alpha \in U$ if and only if
- When you consider $\Bbb{C} = \Bbb{R}^2$, the map $f: U \subset \Bbb{R}^2 \to \Bbb{R}^2$ is $\Bbb{R}$-differentiable in the standard real-multi-variable sense, and
- $f$ satisfies the Cauchy-Riemann equations (so that the real derivative $Df_{\alpha}$ is actually a $\Bbb{C}$-linear map)
But the function $f$ you have, is $f(x,y) = (\sqrt{|xy|}, 0)$, and this is not even real-differentiable as a map from an open subset of $\Bbb{R}^2$ into $\Bbb{R}^2$. Why? Notice that $f$ is positively homogeneous, which means for every $t\geq 0$, we have $f(tx,tx) = t f(x,y)$. By this answer, if $f$ was $\Bbb{R}$-differentiable at the origin, it would have to be an $\Bbb{R}$-linear map, which it clearly isn't.
By the way, you should really take note of the statement proven in the linked answer, because it's often a quick way to prove that several functions defined on $\Bbb{R}^2$ are not $\Bbb{R}$-differentiable at the origin (for example, things like $g(x,y) = \dfrac{x^3}{x^2 + y^2}$ if $(x,y) \neq (0,0)$, and $g(0,0) = 0$ else, or other functions like this are positively homogeneous, but non-linear, hence not $\Bbb{R}$-differentiable). Once you know that theorem, the non-differentiability of such functions becomes almost obvious by inspection.
That statement is false, unless you add something about $\varepsilon$, such that $\lim_{z\to z_0}\varepsilon(z)=0$. Then, asserting that $f$ is differentiable at $z_0$ means that the limit $\lim_{h\to0}\frac{f(z_0+h)-f(z_0)}h$ exists. But\begin{align}\lim_{h\to0}\frac{f(z_0+h)-f(z_0)}h&=\lim_{t\to0}\frac{Ah+\varepsilon(z_0+h)h}h\\&=A+\lim_{h\to0}\varepsilon(z_0+h)\\&=A.\end{align}
Best Answer
$\lim_{h\to 0}|\frac{{\overline h}^2}{h}|=\lim_{h\to 0}\frac{|h|^2}{|h|}= \lim_{h\to 0}|h|=0.$
Thus $\lim_{h\to 0}\frac{{\overline h}^2}{h}=0.$