For the case when $a,b>0,$ I used AM-GM Inequality as follows that:
$\frac{(a+b+\frac{1}{ab})}{3} \geq (ab\frac{1}{ab})^\frac{1}{3}$
This implies that $(a+b+\frac{1}{ab})\geq 3$. Hence, the minimum value of $(a+b+\frac{1}{ab})$ is 3
But the answer is $2+\sqrt{2}$ … how is it ?
Best Answer
If you want to use AM-GM for this, you need to ensure that the equality condition can be met along with the constraint, by "balancing coefficients". Illustrated below:
$$a+b+\frac1{ab} = a+b +\frac{1}{2\sqrt2 ab}+\left(1-\frac1{2\sqrt2}\right)\frac1{ab}$$
Now by AM-GM, $$ a+b +\frac{1}{2\sqrt2 ab}\ge \frac{3}{\sqrt2}$$
and for the remaining term we have again $$\left(1-\frac1{2\sqrt2}\right)\frac1{ab} \ge \left(1-\frac1{2\sqrt2}\right)\frac{2}{a^2+b^2} = 2-\frac{1}{\sqrt2}$$
Combining these results, we have $a+b+\dfrac1{ab} \ge 2 + \sqrt2$, with equality iff $a=b=\frac1{\sqrt2}$.
Added: The key here is of course knowing how to split the LHS, which is by noting that if for equality we need $a=b=\dfrac1{k~ab}$ and for the constraint we need $a^2+b^2=1$, what could be the value of $k$. The rest is then easy applications of AM-GM.