The answer is no.
$\mathcal Hom (\mathcal O_X,\omega)\simeq \omega$, but $\omega\otimes \mathcal Hom (\omega, \mathcal O_X)$ is not necessarily reflexive, let alone locally free. However, it is true that
if $X$ is $G_1$, that is, Gorenstein in codimension $1$, then
$$
(\omega\otimes \mathcal Hom (\omega, \mathcal O_X))^{**}\simeq \mathcal O_X.
$$
EDIT: (due to popular demand, here is a better formed statement for the CM is not even needed part of the original)
This is actually true in a little more general setting:
For simplicity assume that $X$ is $G_1$, $S_2$, equidimensional of dimension $d$ and admits a dualizing complex denoted by $\omega_X^\bullet$. (If, say, by variety you mean a quasi-projective (reduced) scheme of finite type over a field, then the last assumption is automatic. If in addition you also mean irreducible, then so is the equidimensionality. CM obviously implies $S_2$.)
Let
$$\omega_X := h^{-d}(\omega_X^\bullet)$$
and
$$\omega_X^*:=\mathcal Hom_X (\omega_X, \mathcal O_X).$$
Then
$${(\omega_X\otimes \omega_X^*)}^{**}\simeq \mathcal O_X.$$
This follows by the fact that $X$ is $S_2$, both sides are reflexive and they agree in codimension $1$ due to the $G_1$ assumption.
EDIT2: (inspired by Karl's answer):
This actually also implies that
$$\mathcal Hom_X(\omega_X,\omega_X)\simeq \mathcal O_X$$
(under the same conditions) since on the open set where $\omega_X$ is a line bundle,
$$\mathcal Hom_X(\omega_X,\omega_X)\simeq {\omega_X\otimes \omega_X^*}$$
and then since they are both reflexive and $X$ is $S_2$,
$$\mathcal Hom_X(\omega_X,\omega_X)\simeq ({\omega_X\otimes \omega_X^*})^{**}\simeq \mathcal O_X.$$
This works directly when one of $F, G$ is locally free. I am not sure whether it is true when both are merely assumed to be coherent (e.g. I don't see how to get the map). (In general, even the generalization of Serre duality -- Grothendieck duality -- tells you how to hom out of $\mathbf{R}\Gamma \mathcal{F}$ (or more generally derived push-forward) of a sheaf $\mathcal{F}$ into some complex of abelian groups, and this doesn't seem to tell you about $\mathrm{Ext}$ functors up top, in $X$, though perhaps I'm missing something). See below for the extension without local freeness hypotheses.
Namely, there is a map $H^n(X, \omega) \to k$ (the "integration" map*). To get the map
$$\mathrm{Ext}^i(F, G) \to \mathrm{Ext}^{n-i}(G, F \otimes \omega)^*$$ (which is natural), we need a pairing
$$\mathrm{Ext}^i(F, G) \times \mathrm{Ext}^{n-i}(G, F \otimes \omega) \to k.$$
To do this, we can use the Yoneda product to pair these to $\mathrm{Ext}^n(F, F \otimes \omega)$. If $F$ is locally free, then this naturally maps to $\mathrm{Ext}^n(O_X, F \otimes F^{\vee} \otimes \omega)$, which in turns maps to $H^n(X, \omega)$ (by coevaluation) and thus to $k$. If $G$ is locally free, we can similarly write both sides as $\mathrm{Ext}^i(F \otimes G^{\vee}, \omega)$ and $\mathrm{Ext}^{n-i}(O_X, G^{\vee} \otimes F \otimes \omega)^*$, and we get the pairing and isomorphism just as in Hartshorne.
Now if we fix one of $F, G$, we get a $\delta$-functor in the other. So if the natural transformation is an isomorphism when both are locally free, it is an isomorphism when one is locally free and the other merely coherent (since on a projective scheme, every coherent sheaf has a locally free presentation, and we can use the "finite presentation trick").
*Here the comparison is as follows: on a compact complex manifold $X$ of dimension $n$, if $\omega$ denotes the sheaf of holomorphic $(n,0)$-forms, we have (Dolbeaut isomorphism) $$H^n(X, \omega) = \frac{(n,n)\mathrm{-forms}}{\overline{\partial}\mathrm{-exact\ top forms}}$$ and so we can define the map as integration, legitimately (because a $\overline{\partial}$-exact top form is exact in the usual sense, this is well-defined).
Edit: As above, the obstacle to defining the map was that there was no natural trace morphism $$\mathrm{Ext}^n(F, F \otimes \omega) \to H^n(X, \omega);$$ given one, the same arguments would answer your question for the case of $F, G$ both only assumed coherent. As Donu Arapura observes below, there is a natural way to define the trace. Reason: $F$ can be replaced by a bounded complex of locally frees in the derived category (it is a "perfect" complex) since we are working over a smooth variety (in particular, this means that any locally free resolution can be truncated at a finite stage to still yield a locally free one, by Serre's theorem on the finiteness of global dimension). For a bounded complex of locally frees $K^\bullet$, we can define a map
$$\mathrm{Ext}^n(K^\bullet, K^\bullet \otimes \omega) \to H^n(X, \omega)$$
by taking the "partial trace." One can think of the former as consisting of maps $K^\bullet \to K^\bullet \otimes \omega[n]$, or $\mathbf{R}\underline{Hom}(K^\bullet, K^\bullet) \to \omega[n]$. (By the conditions on $K^\bullet$, the derived internal hom is the same as the usual sheaf hom.)
Since there is a natural map from $\mathcal{O}_X$ to the derived internal hom (given by the identity), we can define the trace. To show that map you are interested in becomes an isomorphism, we use the same "finite presentation" trick.
Best Answer
The proof that if $f \colon X \to Y$ is proper and smooth then $f^!\mathcal{O}_Y \cong \Omega^n_F[n]$ ($n=$ dim. rel.$(f)$) assuming only the existence of $f^!$ follows from theorem 3 in
Verdier, Jean-Louis Base change for twisted inverse image of coherent sheaves. 1969 Algebraic Geometry (Internat. Colloq., Tata Inst. Fund. Res., Bombay, 1968) pp. 393–408 Oxford Univ. Press, London.
The proof relies on the "fundamental local isomorphism" and several formal properties of $f^!$. Hope it helps.