There are at least some counterexamples, see equation $(\circ \circ)$ at the bottom and the following discussion.
First I will adress the diophantine equation of the question, though it turned out not to be necessary for the solution of the general problem proposed below, because it is an interesting variant of the solution of the equation for Pythagorean tripels.
I managed to find one particular solution relevant to the original problem and an abundance of solutions for the diophantine equation as such in the case $m \ = \ -1$. In this particular case only, $(4mb^2 - 1)^2$ may be regarded as the square of a sum of an even and an odd square, so it can be expressed as the side of the well-known parametrization equation of Pythagorean triples, which contains only one square. By this Pythagorean parametrization, we can then regard it as the sum of two new squares and write down a solution for the diophantine equation as such:
$$ (u + v)^2 = u^2 + v^2 + 2uv \, ,$$
where $u$, $v$ are integers, where one, say $u$, is even and the other, therefore $v$, is odd and $u,v$ may be multiplied each by another arbitrary integer to obtain yet more solutions. To solve our original problem, we have to account for our special circumstances, which forces the following choices, where we integrate $m = -1$ from the start:
$$ u^2 + v^2 = (1 + 4b^2) \, , \ v = (a^2 + b^2 + 1)^2 \, , \ l = 2u \, ,$$
which in turn lead to a consistency condition for $u$ expressible as the following equation:
$$u^2 \ = \ -a^4 - 2a^2 - (2a^2 - 6)b^2 + 15b^4 \, .$$
The particular solution I found to this one is $a = b = 1$, leading to $u = 4$, which gives $49 = 25 + 24$ as solution to our diophantine equation according to the given parametrization. This solution does not solve our original equation though, so it does not lead to $N(f(x^2))$ being a square. After all, the condition was indeed only necessary, not sufficient. I left the matter there to look for a different approach, which follows.
From now on $N(x) \, = \, N$ and $D(x) \, = \, D$ will be used, because there is no ambiguity in their use below and it improves readability in the long equations.
If we add $(N + 1)(N + 3)$ to the last equation in the question and rearrange, converting some of the terms along the way and putting everything except for $K^2$ on the right hand side, we get:
$$K^2 \ = \ (D-1)^2 \, + \, (N + 3)(D - 1)(N + 1) \, + \, (N^2 + 3)(N + 1)^2 \, . $$
The right hand side is now a not quite quadratic form in $D - 1$ and $N + 1$, which shows how to proceed. We use $N + 3 \, = \, N + 1 + 2$ and $N^2 + 3 = N^2 + 2 + 1$ to rearrange again and obtain
$$K^2 \ = \ (D+N)^2 \, + \, (N^2 + D + 1)(N + 1)^2 \, . $$
We can now turn the right hand side into a square by adding $(D + 4N)(N + 1)^2$, giving
$$K^2 \, + \, (D + 4N)(N + 1)^2 \ = \ (D + N + (N + 1)^2)^2 \, . \ (\circ)$$
At this point, it is expedient to introduce the trace $T(x) \, = \, T \, = \, 2a$ of the algebraic number, as this shows, that we have two squares on the left and shows the special position of the conditions leading to squares from the question. We have $T^2 \, = \, D + 4N$ by the general definitions of norm, trace and discriminant of numbers from a quadratic field, so we get
$$K^2 \, + \, T^2(N + 1)^2 \ = \ (D + (T^2 - D)/4 + (N + 1)^2)^2 \, . \ (\circ \circ) $$
If $T \, = \, 0 \, = \, a$ or $N \, = \, -1$, we obviously always get a true equation in integers, and the case $D \, = \, 0 \, = \, b$ is dealt with affirmatively by putting $c \, = \, 1$, $u \, = \, T/2$ and $v \, = \, N + 1$ in the parametrization of the Pythagorean Triples.
When fitting the case $N \, = \, 1$ into the Pythagorean framework, I found a very small amount of counterexamples, but there may be more under other circumstances:
For $N \, = \, 1$ the choice of $c \, = \, 1, \ u \, = \, a(N + 1), \ v \, = \, 1$ is appropriate to fulfill equation $(\circ \circ)$, in this case the following equation has to be true:
$$ 1 \, + \, a^2 (N+1)^2 \ = \ 3mb^2 \, + \, a^2 \, + \, (N+1)^2 \, , $$
where the right hand side is just one of many ways to write the right hand side of $(\circ \circ)$. Using $mb^2 \, = \, a^2 - N$, we can show, that this is equivalent to
$$ (a^2 - 1)(N+3)(N-1) \ = \ -3(N-1) \, .$$
Here it is obvious, that the equation is always fulfilled for $N \, = \, 1$, but there are some additional solutions: Dividing by $N-1$, we find $N \, = \, -4$ and $a^2 \, = \, 4$ for possible integer solutions, which leads to $8 \, = \, mb^2$. As $m$ is squarefree, this equation has the solutions $m \, = \, 2 \, = \, |b|$.
We therefore find, that $m \, = \, |a| \, = \, |b| \, = \, 2$ leads to the solution $K \, = \, 35$ of our original problem, corresponding to the Pythagorean Triple $35^2 + 12^2 \, = \, 37^2$.
As there is potentially more flexibility in the choice of $c, u, v$, there may still be more counterexamples. Anyway, I have found a sufficiently satisfactory answer for my original question.
Now I have nearly clearing everything up. Let me write them as an answer.
Problem: Let $K$ be a number field and $p$ be prime number. Let $M$ be the maximal abelian pro-$p$ extension of $K$ unramified outside $p$. Describe $\mathrm{Gal}(M|K)$ and calculate its $\mathbb{Z}_p$-rank.
For any modulus $\mathfrak{m}$, we have the exact sequence (given in this post)
$$
0 \to (\mathcal O_K/\mathfrak m)^{\times}/\mathcal O_K^{\times} \xrightarrow{\alpha} \mathrm{Cl}_{\mathfrak m} \to \mathrm{Cl}_K \to 0 \quad\quad (\dagger).
$$
Here the map $\alpha$ can be regarded as the composition
$$
\mathcal O_K^{\times} \xrightarrow{\alpha^{\flat}} (\mathcal O_K/\mathfrak m)^{\times} \rightarrow (\mathcal O_K/\mathfrak m)^{\times} / \mathrm{image}(\alpha^{\flat}).
$$
Now consider the modulus $\mathfrak{m}_n := p^n$ in $K$ for each integer $n$. Let $H_{p,n}$ be the class field of $\mathfrak{m}_n$ over $K$. Then $\mathrm{Cl}_{\mathfrak{m}_n} \cong \mathrm{Gal}(H_{p,n} | K)$ and every places outside the ones lying above $p$ in $K$ is unramified in $H_{p,n}$. Hence $(\dagger)$ becomes
$$
0 \to (\mathcal O_K/(p^n \mathcal{O}_K))^{\times}/\mathcal O_K^{\times} \to \mathrm{Gal}(H_{p,n} | K) \to \mathrm{Cl}_K \to 0
$$
Then we take the $p$-primary part of each object in the exact sequence and get
$$
0 \to ((\mathcal O_K/(p^n \mathcal{O}_K))^{\times}/\mathcal O_K^{\times})[p^{\infty}] \to \mathrm{Gal}(H_{p,n}| K)[p^{\infty}] \to \mathrm{Cl}_K[p^{\infty}] \to 0 \quad\quad (\star)
$$
Here firstly, the functor $[p^{\infty}]$ is left exact since it is the right adjoint of the inclusion functor from the category of $p$-primary abelian groups to the category of abelian groups. Then shrinking the category of abelian groups to torsion abelian groups, we see that it is right exact as well. (This follows from the first comment by @Mindlack, or by the observation that in the category of torsion abelian groups, $-[p^{\infty}] = -\otimes_{\mathbb{Z}} \mathbb{Z}_p$, the latter one is right exact. See this post.)
Then we take the inverse limit in the exact sequence $(\star)$ and get
$$
0 \to \lim ((\mathcal O_K/(p^n \mathcal{O}_K))^{\times}/\mathcal O_K^{\times})[p^{\infty}] \to \lim (\mathrm{Gal}(H_{p,n}| K)[p^{\infty}]) \to \lim \mathrm{Cl}_K[p^{\infty}] \to 0 \quad\quad (\star\star).
$$
Indeed, the functor $\lim$ is left exact. Moreover, since the leftmost term in $(\star)$ is finite (being included in $\mathrm{Gal}(H_{p,n}| K)[p^{\infty}]$) , it satisfies the Mittag-Leffler condition. Hence $\lim$ is right exact here as well.
And note that $\lim \mathrm{Cl}_K[p^{\infty}] = \mathrm{Cl}_K[p^{\infty}]$, once we write explicity the inverse system out. Hence
$$
0 \to \lim ((\mathcal O_K/(p^n \mathcal{O}_K))^{\times}/\mathcal O_K^{\times})[p^{\infty}] \to \lim (\mathrm{Gal}(H_{p,n}| K)[p^{\infty}]) \to \mathrm{Cl}_K[p^{\infty}] \to 0 \quad\quad (\star\star).
$$
So it remains to deal with the remaining two terms.
- For the $\lim (\mathrm{Gal}(H_{p,n}| K)[p^{\infty}])$, we break it step by step:
Dealing with $\mathrm{Gal}(H_{p,n}| K)[p^{\infty}]$. Note that $G_n := \mathrm{Gal}(H_{p,n}| K)$ is a finite abelian groups, we can decompose it as
$$
G_n \cong G_n[p^{\infty}] \cup_{\ell \neq p} G_n[\ell^{\infty}],
$$
where in this case, $G_n[p^{\infty}]$ is the $p$-Sylow subgroup of $G_n$. Consider the subextension $M_{p,n}$ of $H_{p,n}$ fixed by $\cup_{\ell \neq p} G_n[\ell^{\infty}]$. Then $M_{p,n} | K$ has Galois group isomorphic to $G_n[p^{\infty}]$. This is then the maximal $p$-subextension of $H_{p,n}$. Then all places in $K$ outside the ones lying over $p$ in $K$ are unramified in $M_{p,n}$ as they are in $H_{p,n}$. In this way, we have obtained a maximal $p$-subectension $M_{p,n}$ of $H_{p,n}$ such that every place outside $p$ is unramified, and its Galois group $\mathrm{Gal}(M_{p,n}|K) = \mathrm{Gal}(H_{p,n}| K)[p^{\infty}]$.
Now we take the limit of $\mathrm{Gal}(H_{p,n}| K)[p^{\infty}]$ with respect to $n$, by Galois theory, we obtain that
$$
\lim (\mathrm{Gal}(H_{p,n}| K)[p^{\infty}]) = \lim \mathrm{Gal}(M_{p,n}|K) = \mathrm{Gal}(M|K).
$$
Therefore, the middle term in $(\star\star)$ is merely $\mathrm{Gal}(M|K)$.
- For the limit $\lim ((\mathcal O_K/(p^n \mathcal{O}_K))^{\times}/\mathcal O_K^{\times})[p^{\infty}]$, I turn to the following exact sequence first
$$
\mathcal O_K^{\times} \to (\mathcal O_K/(p^n \mathcal{O}_K))^{\times} \to (\mathcal O_K/(p^n \mathcal{O}_K))^{\times}/\mathcal O_K^{\times} \to 1.
$$
Here we have invoked the convention in the shaded part in this answer. So the sequence may not be left exact.
Then tensoring this with $\mathbb{Z}_p$ over $\mathbb{Z}$ we get
$$
\mathcal O_K^{\times} \otimes_{\mathbb{Z}} \mathbb{Z}_p \to (\mathcal O_K/(p^n \mathcal{O}_K))^{\times}\otimes_{\mathbb{Z}} \mathbb{Z}_p \to ((\mathcal O_K/(p^n \mathcal{O}_K))^{\times}/\mathcal O_K^{\times})\otimes_{\mathbb{Z}} \mathbb{Z}_p \to 1.
$$
Since it is a right exact functor. Again by this post, we see that for torsion groups $M$, $M[p^{\infty}] = M \otimes_{\mathbb{Z}} \mathbb{Z}_p$. So the right two terms can be modified (while $O_K^{\times}$ may not be a torsion group by Dirichlet's unit theorem, it will remain unchanged) as
$$
\mathcal O_K^{\times} \otimes_{\mathbb{Z}} \mathbb{Z}_p \to (\mathcal O_K/(p^n \mathcal{O}_K))^{\times}[p^{\infty}]\to ((\mathcal O_K/(p^n \mathcal{O}_K))^{\times}/\mathcal O_K^{\times})[p^{\infty}] \to 1.
$$
And taking inverse limit, we get
$$
\mathcal O_K^{\times} \otimes_{\mathbb{Z}} \mathbb{Z}_p \to \lim(\mathcal O_K/(p^n \mathcal{O}_K))^{\times}[p^{\infty}] \to \lim ((\mathcal O_K/(p^n \mathcal{O}_K))^{\times}/\mathcal O_K^{\times})[p^{\infty}] \to 1. \quad\quad (\star\star\star)
$$
Again we note that the leftmost term $O_K^{\times} \otimes_{\mathbb{Z}} \mathbb{Z}_p$ satisfies the Mittag-Leffler condition, since it is finite (as the right two terms are). Hence here taking inverse limit preserves right exactness.
So to describe the third term, we shall compute the second term in $(\star\star\star)$. We claim that
$$\lim((\mathcal O_K/(p^n \mathcal{O}_K))^{\times}[p^{\infty}]) = \prod_{\mathfrak p \mid p} U_{\mathfrak p}^{1}, $$
where $U_{\mathfrak p}^{1}$ is the subgroup of $\mathcal{O}_{K, \mathfrak p}^{\times}$ consisting of elements congruent to $1$ modulo $p$.
Proof of the claim: we compute
\begin{align*}
\lim((\mathcal O_K/(p^n \mathcal{O}_K))^{\times}[p^{\infty}])
&= \lim\left( \prod_{\mathfrak{p} | p} \left((\mathcal O_K/(\mathfrak{p}^{e_{\mathfrak{p}}n}))^{\times}[p^{\infty}] \right) \right) \\
&= \prod_{\mathfrak{p} | p} \lim\left((O_K/(\mathfrak{p}^{e_{\mathfrak{p}}n}))^{\times}[p^{\infty}]\right) \\
&= \prod_{\mathfrak{p} | p} \lim((O_K/(\mathfrak{p}^{e_{\mathfrak{p}}n}))^{\times}) [p^{\infty}] \\
&= \prod_{\mathfrak{p} | p} (\lim (O_K/(\mathfrak{p}^{e_{\mathfrak{p}}n})))^{\times} [p^{\infty}] \\
&= \prod_{\mathfrak{p} | p} (\mathcal{O}_{K,\mathfrak{p}})^{\times} [p^{\infty}] \\
&= \prod_{\mathfrak{p} | p} (1+ \varpi_{\mathfrak{p}} \mathcal{O}_{K,\mathfrak{p}})) \\
&= \prod_{\mathfrak{p} | p} U_{\mathfrak{p}}^{1}.
\end{align*}
Here:
- 1st equality: Use the multiplicative version Chinese remainder theorem and interchange $\prod$ and $[p^{\infty}]$,
- 2nd equality: Interchange $\lim$ and $\prod$,
- 3rd equality: Interchange $\lim$ and $[p^{\infty}]$. Since $[p^{\infty}]$ is right adjoint, this can be done (note that the resulting $\lim$ does not escape from the category of $p$-primary abelian groups),
- 4th equality: Interchange $\lim$ and $(-)^{\times}$. Note that the functor $(-)^{\times}$ from the category of commutative rings to abelian groups is right adjoint to the functor taking any abelian group $A$ to its group ring $\mathbb{Z}[A]$. So it indeed preserves $\lim$,
- 5th equality: From the definition of completion. Note that the ramification index $e_{\mathfrak{p}} := e(\mathfrak{p} | p)$ will not affect the result.
The rest two equalities follow from definitions. So the claim is proved.
So to sum up, we obtain
$$
1 \rightarrow \prod_{\mathfrak p \mid p} U_{\mathfrak p}^{1} / (O_K^{\times} \otimes_{\mathbb{Z}} \mathbb{Z}_p) \rightarrow \mathrm{Gal}(M|K) \rightarrow \mathrm{Cl}_K[p^{\infty}] \rightarrow 1.
$$
and
$$
O_K^{\times} \otimes_{\mathbb{Z}} \mathbb{Z}_p \rightarrow \prod_{\mathfrak p \mid p} U_{\mathfrak p}^{1} \rightarrow \mathrm{Gal}(M|K) \rightarrow \mathrm{Cl}_K[p^{\infty}] \rightarrow 1.
$$
There are no major distinction among the two. The first one is neater and the second one is helpful when calculating $\mathbb{Z}_p$-rank and concrete examples.
Calculating $\mathbb{Z}_p$-ranks: It suffices to wandering around the second exact sequence above. Let $r_p$ be the $\mathbb{Z}_p$-rank of the image of $O_K^{\times} \otimes_{\mathbb{Z}} \mathbb{Z}_p$ in $\prod_{\mathfrak p \mid p} U_{\mathfrak p}^{1}$. Then
$$
\mathrm{rank}_{\mathbb{Z}_p}(\mathrm{Gal}(M|K)) = \mathrm{rank}_{\mathbb{Z}_p}(\prod_{\mathfrak p \mid p} U_{\mathfrak p}^{1}) - r_p + \mathrm{rank}_{\mathbb{Z}_p}(\mathrm{Cl}_K[p^{\infty}]).
$$
Now, $\mathrm{rank}_{\mathbb{Z}_p}(\mathrm{Cl}_K[p^{\infty}])=0$ since
$$
\mathrm{Cl}_K[p^{\infty}] \otimes_{\mathbb{Z}_p} \mathbb{Q}_p = 0.
$$
Moreover,
\begin{align*}
\left(\prod_{\mathfrak p \mid p} U_{\mathfrak p}^{1}\right) \otimes_{\mathbb{Z}_p} \mathbb{Q}_p
&= \prod_{\mathfrak p \mid p} \left( (1+ \varpi_{\mathfrak{p}} \mathcal{O}_{K,\mathfrak{p}}) \otimes_{\mathbb{Z}_p} \mathbb{Q}_p \right) \\
&\cong \prod_{\mathfrak p \mid p} \left( \mathcal{O}_{K,\mathfrak{p}} \otimes_{\mathbb{Z}_p} \mathbb{Q}_p \right) \\
&\cong \prod_{\mathfrak p \mid p} \left( \mathbb{Z}_p^{e_{\mathfrak{p}}f_{\mathfrak{p}}} \otimes_{\mathbb{Z}_p} \mathbb{Q}_p \right) \\
&=\mathbb{Q}_p^{\sum_{\mathfrak p \mid p}e_{\mathfrak{p}}f_{\mathfrak{p}}} \\
&=\mathbb{Q}_p^{[K:\mathbb{Q}]}.
\end{align*}
Here in the first line, we interchanged $\prod$ and ${-}_{\mathbb{Z}_p} \mathbb{Q}_p$. In the last line, we used the fundamental equality $\sum_{\mathfrak p \mid p}e_{\mathfrak{p}}f_{\mathfrak{p}} = [K:\mathbb{Q}]$. Hence, to sum up,
$$
\mathrm{rank}_{\mathbb{Z}_p}(\mathrm{Gal}(M|K)) = [K:\mathbb{Q}] - r_p,
$$
as desired!
Best Answer
There are two serious errors in your reasoning.
The first is that $a,b$ are fixed, so you can't assume $a-b$ approaches infinity.
Ignoring that error, another serious error is your work on the LHS.
A limit statement of the form $$ \frac{u}{v}\rightarrow 1 $$ does not imply $ u-v\rightarrow 0 $.