Given a state space model of the following form,
$$
\dot{x} = A\,x + B\,u, \tag{1}
$$
$$
y = C\,x + D\,u. \tag{2}
$$
The openloop transfer function of this system can be found by taking the Laplace transform and assuming all initial conditions to be zero (such that $\mathcal{L}\{\dot{x}(t)\}$ can just be written as $s\,X(s)$). Doing this for equation $(1)$ yields,
$$
s\,X(s) = A\,X(s) + B\,U(s), \tag{3}
$$
which can be rewritten as,
$$
X(s) = (s\,I - A)^{-1} B\,U(s). \tag{4}
$$
Substituting this into equation $(2)$ and defining the openloop transfer function $G(s)$ as the ratio between output ($Y(s)$) and input ($U(s)$) yields,
$$
G(s) = C\,(s\,I - A)^{-1} B + D. \tag{5}
$$
In a normal block diagram representation the controller has as an input $r-y$, with $r$ the reference value you would like to have for $y$, and an output $u$, which would be the input to $G(s)$. For now $r$ can be set to zero, so the controller can be defined as the transfer function from $-y$ to $u$.
For an observer based controller ($L$ and $K$ such that $A-B\,K$ and $A-L\,C$ are Hurwitz) for a state space model we can write the following dynamics,
$$
u = -K\,\hat{x}, \tag{6}
$$
$$
\dot{x} = A\,x - B\,K\,\hat{x}, \tag{7}
$$
$$
\dot{\hat{x}} = A\,\hat{x} + B\,u + L(y - C\,\hat{x} - D\,u) = (A - B\,K - L\,C + L\,D\,K) \hat{x} + L\,y. \tag{8}
$$
Similar to equations $(1)$, $(2)$ and $(5)$, the transfer function of the controller $C(s)$, defined as the ratio of $U(s)$ and $-Y(s)$, can be found to be,
$$
C(s) = K\,(s\,I - A + B\,K + L\,C - L\,D\,K)^{-1} L. \tag{9}
$$
If you want to find the total openloop transfer function from "$-y$" to "$y$" you have to keep in mind that in general $G(s)$ and $C(s)$ are matrices of transfer functions, so the order of multiplication matters. Namely you first multiply the error ($r-y$) with the controller and then the plant, the openloop transfer function can be written as $G(s)\,C(s)$. The closedloop transfer function can then be found with,
$$
\frac{Y(s)}{R(s)} = (I + G(s)\,C(s))^{-1} G(s)\,C(s). \tag{10}
$$
It can also be found directly using equations $(2)$ and $(6)$, and the closedloop state space model dynamics,
$$
\begin{bmatrix}
\dot{x} \\ \dot{\hat{x}}
\end{bmatrix} = \begin{bmatrix}
A & -B\,K \\
L\,C & A - B\,K - L\,C
\end{bmatrix} \begin{bmatrix}
x \\ \hat{x}
\end{bmatrix} + \begin{bmatrix}
0 \\ -L
\end{bmatrix} r, \tag{11}
$$
$$
\frac{Y(s)}{R(s)} = \begin{bmatrix}
C & -D\,K
\end{bmatrix} \begin{bmatrix}
s\,I - A & B\,K \\
-L\,C & s\,I - A + B\,K + L\,C
\end{bmatrix}^{-1} \begin{bmatrix}
0 \\ -L
\end{bmatrix}. \tag{12}
$$
Best Answer
The problem is, that you really have two different outputs for $G$. The image is a bit misleading in this regard, because it suggests that both $X$ and $Y$ are outputs of the same $G$.
For state feedback, its often useful to also look at the state space form of a system. For example, $G(s)$ in state space form is
$$ \begin{align} \dot{x} &= A x + B u \\ y &= C x + D u \end{align} $$
Now, instead of this define a state space model that outputs all states:
$$ \begin{align} \dot{x} &= A x + B u \\ y &= x \end{align} $$
This system has the transfer function $G_x(s) = (s I_n - A)^{-1}B$. Note: $I_n$ is a $n \times n$ identity matrix. The output $Y(s)$ is then
$$ Y(s) = C G_x(s) U(s) + D U(s) $$
and so the original transfer function is
$$ G(s) = Y(s)/U(s) = C G_x(s) + D. \tag{1} $$
So you can now rewrite your formula as a function of $G_x$ by using $(1)$ for $G(s)$ and $X(s) = G_x(s) U(s)$.