Wanted to record some observations here...
If we consider the sequence
- $p = a_1 q + r_1$
- $p = a_2(q-r_1) + r_2$
- $p = a_3(q-r_1-r_2) + r_3$
- etc.
and try to solve for the remainders in the form $r_i = c_i p - d_i q$, there is a nice recursive relation:
First, $c_1 = 1$ and $d_1 = a_1$, and in general,
$c_j = 1 + a_j (c_1 + \ldots + c_{j-1})$ and
$d_j = (1+a_1)(1+a_2)\cdots (1+a_{j-1})a_j$
We can also write $c_j(1+a_j) = c_1 + \ldots + c_j$ so that
$c_j = 1 + \frac{a_j}{a_{j-1}}(c_{j-1} (1+a_{j-1})-1)$. This can be expanded further to obtain the form
$c_j = 1 + a_j [ 1 + (1+a_{j-1}) [ 1 + (1+a_{j-2}) [ 1 + \ldots [1 + (1+a_2)]]\ldots]$, or even
$c_j = 1+a_j + a_j(1+a_{j-1}) + a_j(1+a_{j-1})(1+a_{j-2}) + \ldots + a_j(1+a_{j-1})(1+a_{j-2})\cdots(1+a_2)$
Note that the $a_i$ are strictly increasing in the sequence. Suppose the procedure terminates at the $n$-th step (when $r_n=0$). The limit is then $p/a_n$, and $0 = r_n = c_n (p/q) - d_n$ or that $p/q = d_n / c_n$.
I still don't have a closed form expression, but for instance:
For rationals $p/q$ for which the sequence terminates in the first step, $0 = r_1 = c_1 (p/q) - d_1 = p/q - a_1$, so that $q$ is a divisor of $p$, and the limit is $p/q$.
For termination at the second step, $0 = r_2 = c_2 (p/q) - d_2 = (1+a_2)(p/q) - (1+a_1)a_2$, or that $\frac{p}{q} = \frac{(1+a_1)a_2}{1+a_2}$. If there exists $a_1 < a_2$ satisfying this equality, then the sequence terminates in the second step. One such criteria is if $d$ is a divisor of $p$, $q = d+1$ and $p/d - 1 < d$, then the sequence terminates in the second step to $p/d = (1+a_1)$.
For examples: $30/7 = (1+4)6/(1+6)$. Also, $30/4 = 120/16 = (1+7)15/(1+15)$.
Third step termination: $\frac{p}{q} = \frac{ (1+a_1)(1+a_2)a_3 }{ 1 + a_3(1+(1+a_2)) }$, limit is $(1+a_1)(1+a_2)$, and etc.
Also, it appears that if you plug in $a_n = d$ in any formula, you can generate for which $p$ the process converges to $d$ by substituting any $a_1 < a_2 < \ldots < a_{n-1} < d$. Maybe there's more special structure...
First, observe that $f$ as stated is not always well-defined: we don't know its values on rational numbers which are not in the sequence $\left(\frac{p_{n}}{q_{n}}\right)$. I'm going to make the assumption that $\left(\frac{p_{n}}{q_{n}}\right)$ runs uniquely over all rational numbers (which we can do by picking an enumeration of them), so that $f$ is well-defined. Moreover, we can always choose $q_{n} > 0$ (by changing the sign of $p_{n}$ if necessary). We will assume rational numbers are always written in reduced form.
Let $g(x) = \frac{f(x)}{(x-\sqrt{2})^{2}}$. We want to show that $\lim_{x \to \sqrt{2}}g(x)$ exists, and to find its value. Observe that for any irrational number $\alpha \neq \sqrt{2}$, $g(\alpha) = 0$. So, if the limit exists, it should be $0$.
Now, recall Roth's theorem in Diophantine approximation, which states: for any irrational algebraic number $x$ and any real $\lambda> 0$, the inequality
$$\left|x - \frac{p}{q} \right| < \frac{1}{q^{2+\lambda}}$$ has only finitely many coprime integer solutions $(p, q)$. In particular, taking $x = \sqrt{2}$, $\lambda= \frac{1}{4}$, and squaring both sides, we see that only finitely many coprime integer pairs $(p, q)$ do not satisfy
\begin{equation}\tag{1}
\left(\sqrt{2} - \frac{p}{q} \right)^{2} > \frac{1}{q^{4+\frac{1}{2}}}.
\end{equation} Since we have an enumeration $\frac{p_{n}}{q_{n}}$ of the rationals, we can rephrase this as: there exists some $N$ so that for all $n > N$, $(p_{n}, q_{n})$ satisfies (1).
Therefore, for all $n > N$, we have
$$0 < g\left(\frac{p_{n}}{q_{n}}\right) = \frac{f\left(\frac{p_{n}}{q_{n}}\right)}{\left(\frac{p_{n}}{q_{n}}-\sqrt{2}\right)^{2}} = \frac{\frac{1}{q_{n}^{5}}}{\left(\frac{p_{n}}{q_{n}}-\sqrt{2}\right)^{2}} < \frac{\frac{1}{q_{n}^{5}}}{\frac{1}{q_{n}^{4+\frac{1}{2}}}} = \frac{1}{\sqrt{q_{n}}}.$$
Finally, we can conclude. Let $\epsilon > 0$ be arbitrary. There are only finitely many rational numbers $\frac{p_{n}}{q_{n}}$ with $n \leq N$, and none of these are equal to $\sqrt{2}$. There are also only finitely many rational numbers $\frac{p_{n}}{q_{n}}$ with $q_{n} \leq \frac{1}{\epsilon^{2}}$. Therefore, there exists $\delta > 0$ so that $(\sqrt{2}- \delta, \sqrt{2}+\delta)$ contains only rational numbers $\frac{p_{n}}{q_{n}}$ with $n > N$, and $q_{n} > \frac{1}{\epsilon^{2}}.$
For any rational number $\frac{p_{n}}{q_{n}} \in (\sqrt{2}- \delta, \sqrt{2}+\delta) \setminus \{\sqrt{2}\}$, we have
$$0 < g\left(\frac{p_{n}}{q_{n}}\right) < \frac{1}{\sqrt{q_{n}}} < \frac{1}{\sqrt{\frac{1}{\epsilon^{2}}}} = \epsilon.$$ Moreover, by the first computation, for any irrational number $x \in (\sqrt{2}- \delta, \sqrt{2}+\delta) \setminus \{\sqrt{2}\}$, we have
$$g(x) = 0 < \epsilon.$$
So, we see that for any real $x \in (\sqrt{2}- \delta, \sqrt{2}+\delta) \setminus \{\sqrt{2}\}$, we have $|g(x)| < \epsilon.$ Since $\epsilon$ was arbitrary, this proves that
$$\lim_{x \to \sqrt{2}} g(x) = 0,$$ as desired.
Remark: Observe that this proof still works if we replace $\sqrt{2}$ with any irrational algebraic number.
Best Answer
Assume that the polynomial $p(x)$ has three real roots $\alpha_1<\alpha_2<\alpha_3$. Let $r\in\Bbb Q$ such that $\frac{\alpha_1+\alpha_2}2<r<\alpha_2$. Then the largest root, in absolute value, of $q(x)=x^3p(r+1/x)$ is $\beta$ satisfying $\alpha_2=r+1/\beta$.
Write $r=u/v$ with $u,v\in\Bbb Z$ and $v>0$. Then $v^3q(x)$ is an integer coefficients polynomial with leading term $v^3p(0)x^3$. Consequently, $vp(0)\beta$ is the largest root, in absolute value, of a integer coefficients monic polynomial.
A stated in OP, there exists linear recurrence sequences $p_n,q_n$ of degree $3$ (satisfying the same linear recurrence) such that $p_n/q_n\to vp(0)\beta$. Consequently, $$\frac{p'_n}{q'_n}\xrightarrow{n\to\infty} r+1/\beta=\alpha_2$$ where \begin{align} p'_n&=up_n+v^2p(0)q_n\\ q'_n&=vp_n \end{align} which are linear recurrence sequences of degree 3.
As example, $1+2\cos(2\pi/7)$ is the largest root, in absolute value, of the polynomial $x^3-2x^2-x+1$, hence $p_n/q_n\to\cos(2\pi/7)$ where \begin{align} p_{n+3}&=2p_{n+2}+p_{n+1}-p_n&p_0&=b-a&p_1&=c-b&p_2&=c+b-a\\ q_{n+3}&=2q_{n+2}+q_{n+1}-q_n&q_0&=2a&q_1&=2b&p_3&=2c \end{align} for almost every choice of $a,b,c$.