Answering your last question first: A definable homeomorphism is a function which is both definable and a homeomorphism. So yes, a definable homeomorphism (between subsets of an o-minimal structure) is continuous (in the o-minimal topology) and has a continuous inverse, and is therefore bijective.
I would be very surprised if the term "definable homeomorphism" were ever used in a model theory context for anything other than a continuous definable function with continuous inverse, in some setting with a natural topological structure. The topological component also seems crucial to the concept of triangulation, so I'm surprised that you found a paper that defines triangulation in terms of maps that are only definable bijections. Are you sure you've understood that paper correctly?
I looked up the reference to (2.3) in van den Dries's book. He is referring to Definition 2.3 on p. 127, and specifically to the remark that occurs at the end of the definition, at the top of p. 128. It reads:
In that case, each continuous definable $R$-valued function on $\text{cl}(C)$, where $C\in \Phi^{-1}(K)$, has a continuous definable $R$-valued extension to $A$, by lemma (2.2).
I don't see here any requirement that any function be a homeomorphism. So it does seem to apply to the situation you quote in your question, which is about extending a continuous function on the closure of a set up to a continuous function on $A$.
Added, in response to further comments by the OP.
With all due respect to Curry, Ghrist, and Robinson, the way they use the term "definable homeomorphism" is definitely nonstandard, and it seems to me like a really bad idea. For one thing, the word homeomorphism gives entirely the wrong intuition. And for another thing, we already have a perfectly good term: definable bijection.
Now it turns out that in an o-minimal context, every definable bijection $f$ is a piecewise homeomorphism: you can partition the domain and codomain each into finitely many pieces such that $f$ restricts to a homeomorphism between pieces. So maybe Curry, Ghrist, and Robinson want to emphasize this behavior, and suggest that a definable bijection is actually much tamer than an arbitrary bijection. But in this case, it would be better to use van den Dries's term "definable equivalence" (see (2.11) on p. 132) or at least say "definable piecewise homeomorphism".
Again, van den Dries definitely means "definable and continuous with continuous inverse" when he writes "definable homeomorphism", and this is the standard meaning of the term. It seems to me that with this reading, your concerns about the proof of the theorem in van den Dries's book are resolved (and I agree with you that the passage in question requires $\Phi$ to be continuous).
Here is a theme:
If we impose strong enough conditions on definable subsets of $M^1$ (formulas $\varphi(x,\overline{b})$ where $x$ is a single variable), then we can use this to understand definable subsets of $M^n$ for all $n$ (formulas $\varphi(\overline{x},\overline{b})$ where $\overline{x}$ is a tuple of variables).
Here are some examples of "strong enough conditions":
- Minimality: Every definable subset of $M^1$ is finite or cofinite. Equivalently, definable by a quantifier-free formula in the empty language.
- o-minimality: Every definable subset of $M^1$ is a finite union of points and intervals (with respect to a distinguished linear order $<$ in the language). Equivalently, definable by a quantifier-free formula in the language $\{<\}$.
- $C$-minimality: Every definable subset of $M^1$ is definable by a quantifier-free formula in the language $\{C\}$, where $C$ is a distinguished $C$-relation in the language.
- Others, like $P$-minimality, weak o-minimality, etc.
Now for each of the above, we could impose these restrictions on definable sets at one of two levels: We say $M$ is minimal/o-minimal/etc. if the restriction holds for definable subsets of $M^1$. And we say $M$ is strongly minimal/o-minimal/etc. if the restriction holds not just for $M$, but for all models of the complete theory $T = \text{Th}(M)$. Note that the first option is a property of a single structure, while the second is a property of a complete theory.
Sometimes the two notions are equivalent. This happens for o-minimality: Knight, Pillay, and Steinhorn proved that if a structure $M$ is o-minimal, then it is strongly o-minimal. So people don't bother to include the word "strong". On the other hand, minimality is not equivalent to strong minimality (a counterexample is the structure $(\mathbb{N},<)$: it is minimal, but no proper elementary extension is minimal). Strong minimality is more useful than minimality in model theory (this is not so surprising, since we often want to use the compactness theorem and move between models), so you rarely here about minimality without the word "strong" attached.
Historically, strong minimality emerged quite early in model theory, due to its prominant role in the Baldwin-Lachlan theorem in the early 1970s. Much of Shelah's stability theory can be seen as a generalization of the behavior of strongly minimal theories (and yes, every strongly minimal theory is stable).
o-minimality was isolated later, by van den Dries in the mid 1980s. Pillay and Steinhorn named it o-minimality, noting the similarity with the definition of strong minimality. The other minimality notions arose later: once a theme has been enunciated, variations abound.
So the quote by Casanovas you give in the question is pretty straightforward: the definitions have obvious similarities, strongly minimal theories are stable, but o-minimal theories are also tame, despite being unstable (because they include a linear order).
Ok, in what ways are strongly minimal an o-minimal theories similar?
The main way is that they're both instances of the theme: we get a good understanding of definable subsets of $M^n$ for all $n$. In o-minimal theories, this is via the notion of cell decomposition: every definable set is a finite union of particularly simple definable sets called cells. In strongly minimal theories, every definable set has a Morley rank, and it can be decomposed into a finite union of "irreducible" definable sets of that rank. In both settings, we can think of arbitrary definable sets "geometrically", though in the o-minimal case we generalize semialgebraic geometry (polynomial equations and inequalities over $\mathbb{R}$), while in the strongly minimal case we generalize classical algebraic geometry (polynomial equations over $\mathbb{C}$).
Another feature that strongly minimal and o-minimal theories share is that the algebraic closure operator $\text{acl}$ gives rise to a pregeometry. This allows us to assign a natural dimension to each definable set and each model.
In terms of other abstract model theoretic properties, strongly minimal and o-minimal theories also tend to lie on the "tame" side of model-theoretic properties that can be described as saying there is no "randomness" in definable sets. For example, strongly minimal and o-minimal theories are all NIP and dp-minimal and VC-minimal (the latter two properties also have "minimality" in their names, but their definitions don't quite fit the form of the examples I gave above).
One last thing: You explicitly asked in the comments about model completeness and quantifier elimination. Model-theoretic properties come in two flavors: langauge-dependent and language-independent. A language-independent property only refers to the class of definable sets and to the elementary substructure relation. A language-dependent property may refer to the syntax of the formulas defining those sets and to the substructure relation.
Model completeness and quantifier elimination are language-dependent properties. Strong minimality, o-minimality, stability, NIP, etc. are language-independent properties.
You should never expect to have implications between language-independent properties and language-dependent properties: they're just at different levels. A good example to keep in mind to remind yourself of this is the Morleyization construction. If $T$ is any theory (let's say without quantifier elimination), then there is another theory $T'$, the Morleyization of $T$, obtained by adding a new relation symbol $R_\varphi$ for each formula $\varphi$ in the language of $T$ and axioms asserting that $R_\varphi$ is equivalent to $\varphi$. Then $T'$ has the same class of models as $T$ (up to reduct and unique expansion) and models of $T'$ have the same definable sets as the corresponding models of $T$. So $T$ and $T'$ agree on all language-independent properties, but $T'$ has quantifier elimination and $T$ does not.
Best Answer
I have an answer now:
note the following equivalence:
\begin{align} (x_1,\dots , x_m)\in U&\Leftrightarrow \exists y_1\dots\exists y_m\exists z_1\dots\exists z_m \\ [&(y_1<x_1<z_1)\wedge\dots\wedge (y_m<x_m<z_m)] \wedge\ \\ \big[\forall w_1\dots\forall w_m[&(y_1<w_1<z_1)\wedge\dots\wedge (y_m<w_m<z_m) \wedge (w_1,\dots, w_m)\in B] \\ \rightarrow[&(w_1,\dots, w_m)\in A]\big] \end{align}
Therefore, we have that $U$ is a definable open set.