The basic question here is a linear algebra question: which orientation-preserving linear maps $T:\mathbb{R}^3\to\mathbb{R}^3$ preserve solid angles formed by triples of vectors? (You are then asking about smooth maps whose derivative at every point is such a linear map.) Let $G\subset GL_3(\mathbb{R})$ be the group of such maps, and let $H=\mathbb{R}^+\times SO(3)\subset GL_3(\mathbb{R})$ be the group of conformal linear maps (i.e., compositions of rotations and scalings). It is clear that $H\subseteq G$; I claim that in fact $G=H$, so your maps are just the same as conformal maps.
Suppose $T\in G$; we wish to show that $T\in H$. Let $e_1,e_2,e_3$ be the standard basis vectors for $\mathbb{R}^3$. Taking a singular value decomposition of $T$, we may compose $T$ with rotations (which does not change whether $T\in H$) to assume that $T$ is diagonal. Since $T$ is orientation-preserving, we can compose it with a further rotation to assume the diagonal entries are positive, and we can compose with a scaling to assume that $T(e_1)=e_1$. We have $T(e_2)=be_2$ and $T(e_3)=ce_3$ for positive scalars $b$ and $c$; I claim that $b=c=1$ so $T$ is just the identity.
To prove this consider the solid angle formed by the unit vectors $e_1,\frac{e_1+e_2}{\sqrt{2}},$ and $e_3$. $T$ maps these vectors to $e_1,\frac{e_1+be_2}{\sqrt{2}},$ and $ce_3$, which form the same solid angle as the unit vectors $e_1,\frac{e_1+be_2}{\sqrt{1+b^2}},$ and $e_3$. But now observe that as $b$ grows from $0$ to $\infty$, the spherical triangle formed by these vectors grows strictly (we are taking the spherical triangle formed by $e_1,e_2,$ and $e_3$ and replacing the $e_2$ vertex with a point on the edge between $e_1$ and $e_2$). So, the only way the area of this triangle would be the same as the area of the triangle formed by $e_1,\frac{e_1+e_2}{\sqrt{2}},$ and $e_3$ is if $b=1$. Swapping the roles of $e_2$ and $e_3$, we similarly conclude that $c=1$.
Similar arguments show that for any $n\geq m\geq 2$, the orientation-preserving linear maps $T:\mathbb{R}^n\to\mathbb{R}^n$ that preserve "$m$-dimensional solid angles" formed by $m$ vectors are the same as the conformal linear maps.
I assume you are talking about a two-dimensional set $U$. For simplicity, $U\subset\mathbb R^2$, $f\colon U\to \mathbb R^2$ to avoid geometric issues.
Let's say that in the first definition you assume that the function is $C^1$. Take a conformal map $f$ as in the first definition. The fact that $f$ preserves angles can be rewritten as follows (this should be proved of course, but I will skip this):
- The Jacobian of $f$ is invertible at each point.
- The Jacobian of $f$ at each point is an orthogonal matrix multiplied by a scalar.
If you also assume the orientation is preserved, the jacobian of $f$ at every point is of the form
$$ Jf(x,y)=\left[\begin{aligned}a\,\,&b\\-b\,\,&a \end{aligned}\right],\qquad\exists a,b\in\mathbb R.$$
That is, $f$ satisfies the Cauchy-Riemann equations, that is, if $f(x,y)=(u(x,y),v(x,y))$,
$$ \frac{du}{dx}=\frac{dv}{dy},\,\,\,\,\frac{dv}{dx}=-\frac{du}{dy}. $$
This is in fact equivalent to the function $f$ being holomorphic if $f$ is $C^1$.
Edit. So, I guess that the Jacobian is invertible is necessary to keep curves smooth when you transport them through the map $f$ (think about the holomorphic function $z\mapsto z^2$, which maps lines that pass through the origin to angle-shaped figures).
Once your jacobian is invertible, you essentially want that the cosine of two tangent vectors is preserved through the Jacobian. If you impose this to hold for arbitrary two vectors in the tangent plane, you should end up saying that the Jacobian is indeed an orthogonal transfomation (i.e., that preserves the scalar product) possibly multiplied by a non-zero constant: this follows from the fact that orthogonal pairs of vectors are mapped to orthogonal pairs of vectors. This is an exercise of linear algebra, I don't know how a reference for this sorry (it works in any dimension by the way).
Once you have that, in the two dimensional case it is easy to classify all orthogonal matrices that preserve orientation:
$$ A=\left[\begin{aligned}\cos(\theta)\,\,&\sin(\theta)\\-\sin(\theta)\,\,&\cos(\theta) \end{aligned}\right],\qquad\exists \theta\in\mathbb R.$$
So when you allow for multiplication by a constant, you end up with the above ansatz.
Best Answer
The answer is yes. See Theorem 5.2 in this paper