To find the equilibrium points, we want to find the points where $x'_1$ and $x'_2$ are zero.
So, we want to solve:
$\tag 1 x_1-x_2-x_1^3 = 0$
$\tag 2 x_1+x_2-x_2^3=0$
Using $(2)$, we have $x_1 = x_2^3 - x_2$ and substituting this into equation $(1)$ yields:
$$x_2^3 - x_2 - x^2 -(x_2^3 - x_2)^3 = -x_2 (x_2^8-3 x_2^6+3 x_2^4-2 x_2^2+2) = 0$$
- One solution is $x_2=0$.
- The other solutions for the eighth-order polynomial are all imaginary. I used this Wolfram Alpha solution for the roots, but you should be able to use your favorite tool for root finding.
If we substitute these back into $x_1 = x_2^3 - x_2$, we get the corresponding point as $0$. So, our critical point is $(0,0)$.
Now, you can use these two points with the Jacobian (assuming they are well behaved, do you know when you can use linearization and when you cannot) and determine how these equilibria behave.
Hint: Look at these two points in the phase portrait and your analysis should show this.
I will just give a sketch for now how to solve it, I can fill in more details if necessary.
First define vectors $v_i = (x_i-a,y_i-b),\, i = 1,2,3$. Then we can rewrite the equations as
\begin{align}
\langle v_1, v_2\rangle = a_{12},\ \langle v_2, v_3\rangle = a_{23},\ \langle v_3, v_1\rangle = a_{31},\\
\det[v_1^t\ v_2^t] = b_{12},\ \det[v_2^t\ v_3^t] = b_{23},\ \det[v_3^t\ v_1^t] = b_{31},\\
\end{align} where I changed dot's and det's to a's and b's.
Denote by $\vartheta_{ij}$ the (oriented) angle between vectors $v_i$ and $v_j$. Remember that dot product in $\mathbb R^2$ is given by formula $\|v\|\|w\|\cos\vartheta$ and that determinants above measure (signed) area of parallelogram formed by the vectors, which can also be expressed as $\|v\|\|w\|\sin\vartheta$, and thus the system becomes
\begin{align}
\|v_1\|\|v_2\|\cos\vartheta_{12} = a_{12},\ \|v_2\|\|v_3\|\cos\vartheta_{23} = a_{23},\ \|v_3\|\|v_1\|\cos\vartheta_{31} = a_{31},\\
\|v_1\|\|v_2\|\sin\vartheta_{12} = b_{12},\ \|v_2\|\|v_3\|\sin\vartheta_{23} = b_{23},\ \|v_3\|\|v_1\|\sin\vartheta_{31} = b_{31}.\\
\end{align}
Then, you can find the lengths $\|v_i\|$ by looking at the system $$\|v_1\|\|v_2\| = c_{12},\ \|v_2\|\|v_3\| = c_{23},\ \|v_3\|\|v_1\| = c_{31},$$ where $c_{ij} = \sqrt{a_{ij}^2+b_{ij}^2}$ obtained by squaring the above equations and using $\sin^2t+\cos^2t = 1$. Solving it gives you $$\|v_1\|=\sqrt{\frac{c_{12}c_{31}}{c_{23}}},\ \|v_2\|=\sqrt{\frac{c_{12}c_{23}}{c_{31}}},\ \|v_3\|=\sqrt{\frac{c_{23}c_{31}}{c_{12}}}.$$
The whole thing is, from geometric perspective, obviously rotationally invariant (rotating all of the vectors won't change the angles between them or the areas), so fix some angle $\vartheta$. Then, the solution can be represented as complex numbers as
$$v_1 = \|v_1\|e^{i\vartheta},\ v_2 = \|v_2\|e^{i(\vartheta +\vartheta_{12})},\ v_3 = \|v_3\|e^{i(\vartheta-\vartheta_{31})}.$$
More explicitly, $e^{i\vartheta_{12}} = \cos\vartheta_{12} + i\sin\vartheta_{12} = \frac{a_{12}}{c_{12}}+\frac{b_{12}}{c_{12}}i$ and $e^{i\vartheta_{31}} = \cos\vartheta_{31} + i\sin\vartheta_{31} = \frac{a_{31}}{c_{31}}+\frac{b_{31}}{c_{31}}i$, so we have
\begin{align}v_1 &= \sqrt{\frac{c_{12}c_{31}}{c_{23}}}(\cos\vartheta + i\sin\vartheta),\\
v_2 &= \sqrt{\frac{c_{12}c_{23}}{c_{31}}}(\cos\vartheta + i\sin\vartheta)(\frac{a_{12}}{c_{12}}+i\frac{b_{12}}{c_{12}}),\\
v_3 &= \sqrt{\frac{c_{23}c_{31}}{c_{12}}}(\cos\vartheta + i\sin\vartheta)(\frac{a_{31}}{c_{31}}-i\frac{b_{31}}{c_{31}}),\ \vartheta\in\mathbb R.\end{align}
All you have to do now is expand and $x$'s will be the real parts, while $y$'s the imaginary parts of the above complex number representation.
Note, however, that the system is overdetermined since knowing the angles between $v_1$ and $v_2$ and $v_2$ and $v_3$ will also give you the angle between $v_1$ and $v_3$.
Best Answer
So, I suppose that you have three data points ($i=1,2,3)$ $$y_i=\frac{a}{b+(x_i-c)^2}$$ from which you want to extract parameters $a,b,c$.
If you compute the ratio $A_{12}=\frac{y_2}{y_1}$, $a$ is gone and you can extract $b$. you can do the same with $A_{13}=\frac{y_3}{y_1}$ and extract $b$ again. Write that these two values of $b$ are equal and you obtain a linear equation in $c$ which is easy to solve. Now, go backwards to get successively $b$ and $a$. After simplifications, you should then get
$$c=\frac{1}{2}\frac{{x_1}^2 {y_1} ({y_2}-{y_3})+{x_2}^2 {y_2} ({y_3}-{y_1})+{x_3}^2 {y_3} ({y_1}-{y_2})}{ {x_1} {y_1} ({y_2}-{y_3})+{x_2} {y_2} ({y_3}-{y_1})+{x_3} {y_3} ({y_1}-{y_2})}$$ $$b=\frac{{y_2} (c-{x_2})^2-{y_1} (c-{x_1})^2}{{y_1}-{y_2}}$$ $$a={y_1} \left(b+(c-{x_1})^2\right)$$
Added later
Since this is to be used for regression and since this model is nonlinear with respect to its parameters, linearization would provide reasonable starting guesses for the nonlinear regression.
So, let us start with $$y=\frac{a}{b+(x-c)^2}$$ and rewrite it as $$\frac{1}{y}=\frac{x^2-2cx+(b+c^2)}{a}=\alpha x^2+\beta x+\gamma$$ The quadratic regression based on points$(x_i,\frac{1}{y_i})$ will give $\alpha,\beta,\gamma$. So, now $$a=\frac{1}{\alpha},b=\frac{4 \alpha \gamma -\beta ^2}{4 \alpha ^2},c=-\frac{\beta }{2 \alpha }$$
But again, the values are just starting guesses since in the linearized model you minimize $$SSQ_1=\sum_{i=1}^N \Big(\frac{1}{y_i^{exp}}-\frac{1}{y_i^{calc}}\Big)^2$$ while, since the $y_i$ are the measured values, you must minimize $$SSQ_2=\sum_{i=1}^N \Big({y_i^{exp}}-{y_i^{calc}}\Big)^2$$
Illustrating example
To illustrate this aspect, I generated data using $a=123.4,b=5.6,c=7.8$ for $x$ varying between $0$ and $20$ ($21$ data points) and the exact values of $y$ were randomly perturbed (relative change between $-10$% and $+10$%).
The linearization step leads to $\alpha=8.323492415\times 10^{-3}$,$\beta=-1.299834856\times 10^{-1}$,$\gamma=5.468572881\times 10^{-1}$ to which correspond $a=120.142$, $b=4.732$, $c=7.808$. These do not seem bad when compared to the values used for the generation of the data. However, for these values, the sum of squares is $45.118$ and the nonlinear regression ends with a sum of squares equal to $3.932$ that is to say almost $12$ times smaller and the final parameters are $a=125.559$, $b=5.998$, $c=7.799$. Big differences !