Given a state space model of the following form,
$$
\dot{x} = A\,x + B\,u, \tag{1}
$$
$$
y = C\,x + D\,u. \tag{2}
$$
The openloop transfer function of this system can be found by taking the Laplace transform and assuming all initial conditions to be zero (such that $\mathcal{L}\{\dot{x}(t)\}$ can just be written as $s\,X(s)$). Doing this for equation $(1)$ yields,
$$
s\,X(s) = A\,X(s) + B\,U(s), \tag{3}
$$
which can be rewritten as,
$$
X(s) = (s\,I - A)^{-1} B\,U(s). \tag{4}
$$
Substituting this into equation $(2)$ and defining the openloop transfer function $G(s)$ as the ratio between output ($Y(s)$) and input ($U(s)$) yields,
$$
G(s) = C\,(s\,I - A)^{-1} B + D. \tag{5}
$$
In a normal block diagram representation the controller has as an input $r-y$, with $r$ the reference value you would like to have for $y$, and an output $u$, which would be the input to $G(s)$. For now $r$ can be set to zero, so the controller can be defined as the transfer function from $-y$ to $u$.
For an observer based controller ($L$ and $K$ such that $A-B\,K$ and $A-L\,C$ are Hurwitz) for a state space model we can write the following dynamics,
$$
u = -K\,\hat{x}, \tag{6}
$$
$$
\dot{x} = A\,x - B\,K\,\hat{x}, \tag{7}
$$
$$
\dot{\hat{x}} = A\,\hat{x} + B\,u + L(y - C\,\hat{x} - D\,u) = (A - B\,K - L\,C + L\,D\,K) \hat{x} + L\,y. \tag{8}
$$
Similar to equations $(1)$, $(2)$ and $(5)$, the transfer function of the controller $C(s)$, defined as the ratio of $U(s)$ and $-Y(s)$, can be found to be,
$$
C(s) = K\,(s\,I - A + B\,K + L\,C - L\,D\,K)^{-1} L. \tag{9}
$$
If you want to find the total openloop transfer function from "$-y$" to "$y$" you have to keep in mind that in general $G(s)$ and $C(s)$ are matrices of transfer functions, so the order of multiplication matters. Namely you first multiply the error ($r-y$) with the controller and then the plant, the openloop transfer function can be written as $G(s)\,C(s)$. The closedloop transfer function can then be found with,
$$
\frac{Y(s)}{R(s)} = (I + G(s)\,C(s))^{-1} G(s)\,C(s). \tag{10}
$$
It can also be found directly using equations $(2)$ and $(6)$, and the closedloop state space model dynamics,
$$
\begin{bmatrix}
\dot{x} \\ \dot{\hat{x}}
\end{bmatrix} = \begin{bmatrix}
A & -B\,K \\
L\,C & A - B\,K - L\,C
\end{bmatrix} \begin{bmatrix}
x \\ \hat{x}
\end{bmatrix} + \begin{bmatrix}
0 \\ -L
\end{bmatrix} r, \tag{11}
$$
$$
\frac{Y(s)}{R(s)} = \begin{bmatrix}
C & -D\,K
\end{bmatrix} \begin{bmatrix}
s\,I - A & B\,K \\
-L\,C & s\,I - A + B\,K + L\,C
\end{bmatrix}^{-1} \begin{bmatrix}
0 \\ -L
\end{bmatrix}. \tag{12}
$$
Best Answer
Based on the comment, this is a periodic system that can be written as
$$x(3(k+1))=A^2(A+BK)x(3k).$$
Asymptotic stability is ensured provided that the eigenvalues of $A^2(A+BK)$ are inside the unit disc. A way to ensure that is through the consideration of a quadratic Lyapunov function of the form $V(x)=x^TPx$ where $P$ symmetric positive definite (i.e. $P\succ0$). This yields the following Linear Matrix Inequality
$$\begin{bmatrix} -Q & A^3Q+A^2BU\\(A^3Q+A^2BU)^T & -Q \end{bmatrix}\prec0$$
where $U$ and $Q\succ0)$ are matrices to be computed and the inequality sign means that the above expression is negative definite. A suitable gain $K$ can be constructed using the expression $K=UQ^{-1}$.
This can be slightly modified by considering a scalar $\beta\in(0,1)$ and solving for the condition
$$\begin{bmatrix} -\beta^2 Q & A^3Q+A^2BU\\(A^3Q+A^2BU)^T & -Q \end{bmatrix}\prec0,$$
which is equivalent to saying that the spectral radius of $A^2(A+BK)$ is less than $\beta$.
A more direct approach (see Kwin's comment) is to just design a state-feedback controller for the system $(A^3,A^2B)$. It is interesting to note that this system may not be controllable even if $(A,B)$ is. A simple example is the integrator $\dot{x}=u$.