Given a state space model of the following form,
$$
\dot{x} = A\,x + B\,u, \tag{1}
$$
$$
y = C\,x + D\,u. \tag{2}
$$
The openloop transfer function of this system can be found by taking the Laplace transform and assuming all initial conditions to be zero (such that $\mathcal{L}\{\dot{x}(t)\}$ can just be written as $s\,X(s)$). Doing this for equation $(1)$ yields,
$$
s\,X(s) = A\,X(s) + B\,U(s), \tag{3}
$$
which can be rewritten as,
$$
X(s) = (s\,I - A)^{-1} B\,U(s). \tag{4}
$$
Substituting this into equation $(2)$ and defining the openloop transfer function $G(s)$ as the ratio between output ($Y(s)$) and input ($U(s)$) yields,
$$
G(s) = C\,(s\,I - A)^{-1} B + D. \tag{5}
$$
In a normal block diagram representation the controller has as an input $r-y$, with $r$ the reference value you would like to have for $y$, and an output $u$, which would be the input to $G(s)$. For now $r$ can be set to zero, so the controller can be defined as the transfer function from $-y$ to $u$.
For an observer based controller ($L$ and $K$ such that $A-B\,K$ and $A-L\,C$ are Hurwitz) for a state space model we can write the following dynamics,
$$
u = -K\,\hat{x}, \tag{6}
$$
$$
\dot{x} = A\,x - B\,K\,\hat{x}, \tag{7}
$$
$$
\dot{\hat{x}} = A\,\hat{x} + B\,u + L(y - C\,\hat{x} - D\,u) = (A - B\,K - L\,C + L\,D\,K) \hat{x} + L\,y. \tag{8}
$$
Similar to equations $(1)$, $(2)$ and $(5)$, the transfer function of the controller $C(s)$, defined as the ratio of $U(s)$ and $-Y(s)$, can be found to be,
$$
C(s) = K\,(s\,I - A + B\,K + L\,C - L\,D\,K)^{-1} L. \tag{9}
$$
If you want to find the total openloop transfer function from "$-y$" to "$y$" you have to keep in mind that in general $G(s)$ and $C(s)$ are matrices of transfer functions, so the order of multiplication matters. Namely you first multiply the error ($r-y$) with the controller and then the plant, the openloop transfer function can be written as $G(s)\,C(s)$. The closedloop transfer function can then be found with,
$$
\frac{Y(s)}{R(s)} = (I + G(s)\,C(s))^{-1} G(s)\,C(s). \tag{10}
$$
It can also be found directly using equations $(2)$ and $(6)$, and the closedloop state space model dynamics,
$$
\begin{bmatrix}
\dot{x} \\ \dot{\hat{x}}
\end{bmatrix} = \begin{bmatrix}
A & -B\,K \\
L\,C & A - B\,K - L\,C
\end{bmatrix} \begin{bmatrix}
x \\ \hat{x}
\end{bmatrix} + \begin{bmatrix}
0 \\ -L
\end{bmatrix} r, \tag{11}
$$
$$
\frac{Y(s)}{R(s)} = \begin{bmatrix}
C & -D\,K
\end{bmatrix} \begin{bmatrix}
s\,I - A & B\,K \\
-L\,C & s\,I - A + B\,K + L\,C
\end{bmatrix}^{-1} \begin{bmatrix}
0 \\ -L
\end{bmatrix}. \tag{12}
$$
Well, when we have a transfer function that looks like:
$$\mathcal{T}:=\frac{\text{Y}\left(\text{s}\right)}{\text{X}\left(\text{s}\right)}=\frac{\text{G}\left(\text{s}\right)}{1+\text{G}\left(\text{s}\right)\text{H}\left(\text{s}\right)}\tag1$$
Where $\text{H}\left(\text{s}\right):=1$ because of the unity feedback and $\text{G}\left(\text{s}\right):=\text{K}\cdot\frac{1}{\text{s}-\alpha}$
We get a complete transfer function that looks like:
$$\mathcal{T}=\frac{\text{Y}\left(\text{s}\right)}{\text{X}\left(\text{s}\right)}=\frac{\text{K}\cdot\frac{1}{\text{s}-\alpha}}{1+\text{K}\cdot\frac{1}{\text{s}-\alpha}}\tag2$$
Now, for the poles and zeros:
- Poles:
$$1+\text{K}\cdot\frac{1}{\text{s}-\alpha}=0\space\Longleftrightarrow\space\text{s}=\alpha-\text{K}\tag3$$
- Zeros:
$$\text{K}\cdot\frac{1}{\text{s}-\alpha}=0\space\Longleftrightarrow\space\text{K}=0\tag4$$
For the settling-time we have that:
$$t\space_{2\text{%}}=-\frac{\ln\left(50\right)}{\lambda}\space\Longleftrightarrow\space\lambda=-\frac{\ln\left(50\right)}{t\space_{2\text{%}}}\tag5$$
We need to use:
$$\text{s}=\lambda+\omega\text{j}\tag6$$
Where $\lambda\space\wedge\space\omega\in\mathbb{R}$ and $\text{j}^2=-1$
Now, we need to solve for $\text{K}$ in the pole equation (assuming $\text{K}\ne0$):
$$1+\text{K}\cdot\frac{1}{\left(-\frac{\ln\left(50\right)}{t\space_{2\text{%}}}+\omega\text{j}\right)-\alpha}=0+0\text{j}\space\Longleftrightarrow\space\text{K}=\alpha+\frac{\ln\left(50\right)}{t\space_{2\text{%}}}\space\space\space\wedge\space\space\space\omega=0\tag7$$
Conclusion, we get a transfer function that looks like:
$$\color{red}{\mathcal{T}=\frac{\text{Y}\left(\text{s}\right)}{\text{X}\left(\text{s}\right)}=\frac{\alpha\cdot t\space_{2\text{%}}+\ln\left(50\right)}{\text{s}\cdot t\space_{2\text{%}}+\ln\left(50\right)}}\tag8$$
Best Answer
You have to split your closed loop in the linear part and the nonlinear part.
You have already done it in your below picture. For computing the linear transfer function (LTI), just delete the nonlinear block and simplify the rest of blocks choosing as input and output where the nonlinear block is attached to the linear part. In your picture, $x$ should be the output and the input should be the input to the first integrator.
After this, you will have two blocks, one linear (represented by your LTI), and the other nonlinear block.
You can use the well known results for LTI systems about one operation point, but not for every $x$.