Continuing where you stopped: Since $f$ is bijective, we have that $f(f^{-1}(U)) = U$ and since $U$ is open, we have that $f(V)$ is open (in $Y$)
To prove that $f^{-1}$ is continuous, we should prove that for all open sets $U \in X$, $(f^{-1})^{-1}(U)$ is open in $Y$, since $(f^{-1})^{-1}(U) = f(U)$. And since $f$ is open, this follows directly.
Answering your last question first: A definable homeomorphism is a function which is both definable and a homeomorphism. So yes, a definable homeomorphism (between subsets of an o-minimal structure) is continuous (in the o-minimal topology) and has a continuous inverse, and is therefore bijective.
I would be very surprised if the term "definable homeomorphism" were ever used in a model theory context for anything other than a continuous definable function with continuous inverse, in some setting with a natural topological structure. The topological component also seems crucial to the concept of triangulation, so I'm surprised that you found a paper that defines triangulation in terms of maps that are only definable bijections. Are you sure you've understood that paper correctly?
I looked up the reference to (2.3) in van den Dries's book. He is referring to Definition 2.3 on p. 127, and specifically to the remark that occurs at the end of the definition, at the top of p. 128. It reads:
In that case, each continuous definable $R$-valued function on $\text{cl}(C)$, where $C\in \Phi^{-1}(K)$, has a continuous definable $R$-valued extension to $A$, by lemma (2.2).
I don't see here any requirement that any function be a homeomorphism. So it does seem to apply to the situation you quote in your question, which is about extending a continuous function on the closure of a set up to a continuous function on $A$.
Added, in response to further comments by the OP.
With all due respect to Curry, Ghrist, and Robinson, the way they use the term "definable homeomorphism" is definitely nonstandard, and it seems to me like a really bad idea. For one thing, the word homeomorphism gives entirely the wrong intuition. And for another thing, we already have a perfectly good term: definable bijection.
Now it turns out that in an o-minimal context, every definable bijection $f$ is a piecewise homeomorphism: you can partition the domain and codomain each into finitely many pieces such that $f$ restricts to a homeomorphism between pieces. So maybe Curry, Ghrist, and Robinson want to emphasize this behavior, and suggest that a definable bijection is actually much tamer than an arbitrary bijection. But in this case, it would be better to use van den Dries's term "definable equivalence" (see (2.11) on p. 132) or at least say "definable piecewise homeomorphism".
Again, van den Dries definitely means "definable and continuous with continuous inverse" when he writes "definable homeomorphism", and this is the standard meaning of the term. It seems to me that with this reading, your concerns about the proof of the theorem in van den Dries's book are resolved (and I agree with you that the passage in question requires $\Phi$ to be continuous).
Best Answer
As Kavi Rama Murthy already said in the comments, yes $(f^{-1})^{-1}(U)=f(U)$. I will give you a simple hint on how to prove it. By definition: $$(f^{-1})^{-1}(U)=\{y\in Y : f^{-1}(y)\in U\}$$ And finally $f^{-1}(y)\in U \iff y=f(x)$ for $x\in U$ (again, by definition). I think you can prove the rest.
However, note that your proof is not complete if you stop here, since you want to prove that those are equivalent definitions for homeomorphism. You have just proved the following: $$[f\colon X\to Y\ \text{is a homeomorphism}] \implies [U \text{ is open in } X \iff f(U) \text{ is open in } Y]$$ Now, you would have to prove that the converse is also true. As an important remark (you didn't state this clearly), the bijectivity of $f$ has to be assumed in both directions, i.e., the two statements are equivalent when previously assuming that $f$ is a bijection. That means that you have to prove that: $$[f:X\to Y \text{ is bijective and }U \text{ is open in } X \iff f(U) \text{ is open in } Y]\implies[f\colon X\to Y\ \text{is a homeomorphism}]$$ I will leave this proof up to you, since you already have all the tools you need. If you get stuck, don't hesitate to write a comment.