Ad-invariant scalar product on complex vector space

Question

Let's say we have a Lie algebra $\mathfrak{g}$ over a complex vector space, $z_1,z_2,z_3 \in \mathfrak{g}$. What is the defining property for an ad-invariant real valued scalar product
\begin{equation}
\langle z_1,[z_2,z_3] \rangle = \langle [z_1,z_2^*],z_3 \rangle
\end{equation}
or
\begin{equation}
\langle z_1,[z_2,z_3] \rangle = \langle [z_1,z_2],z_3 \rangle
\end{equation}

Answer 1

First of all, the "correct" definition of invariance for a bilinear or hermitian form $\langle \cdot, \cdot\rangle$ on a vector space $V$ with respect to a Lie algebra representation $\rho: \mathfrak g \rightarrow \mathfrak{gl}(V)$ is:

$$\langle \rho(x).v,w\rangle = -\langle v, \rho(x).w\rangle \qquad\text{ for all } x \in \mathfrak g, v,w \in V. \quad (*)$$

For $\rho=ad$ (and e.g. the Killing form) this is often written in the way $\langle [a,b],c\rangle = \langle a, [b,c]\rangle$ you allude to, which in that case is equivalent to the above via $x=-b, a=v, c=w$ and using $[r,s]=-[s,r]$. Still, the above should be preferred due to its much greater generality.

But also with that better definition, you will notice (analogously to your own comment), you run into a problem if $V$ is complex, $\langle \cdot, \cdot \rangle$ is hermitian (i.e. conjugate-linear in, say, the second components), and if you are allowed to multiply $\rho(x)$ through with complex scalars. Indeed, $(*)$ implies that for all scalars $a$ that you can multiply $x$ with and which commute with $\rho$:

$a\langle \rho(x).v,w\rangle = \langle a\rho(x).v,w\rangle = \langle \rho(ax).v,w\rangle \stackrel{(*)}= -\langle v, \rho(ax).w\rangle = -\langle v, a\rho(x).w\rangle =-\color{red}{\bar a}\langle v, \rho(x).w\rangle$

which divided by the original $(*)$ implies $a=\bar a$.

Now the way out is not to not allow complex $V$ or try to force the hermitian inner product to be "real valued" as in the comments, because that would force it to be identically $0$ (and hence not really a product).

Further, as regards your proposed first option, as several comments point out, there is no natural definition for something like $z^*$ in this setting if $\mathfrak g$ is a complex Lie algebra. Indeed, to define that, you would need to choose a real structure of that $\mathfrak g$. The proposed matrix trick also doesn't work in general: What if the conjugate transpose is not in the Lie algebra? (E.g. just take as $\mathfrak g$ some Lie algebra of upper triangular matrices.)

The actual way out is to realize (pun intended) that all this only works for real Lie algebras $\mathfrak g$, so that in the above, $a \in \mathbb R$. Actually, it should turn out later in the theory that such invariant inner products need a kind of compactness which only works for certain real Lie algebras, not complex ones. This way, the theory is treated in Bourbaki's Lie Groups and Algebras, chapter IX (Compact Lie Groups), which starts with a discussion of exactly this kind of invariant hermitian products.