I'm not sure if I would call this "balancing", but in introducing an auxiliary function $f$ you got a degree of freedom in the collection of functions $k,y,f$ that can be fixed to any functional relation. Additionally you gain a degree of freedom by splitting the time scales. So selecting to set the sum of these two terms to zero fixes, removes one degree of freedom and leaves you with two new equations
$$
f^2_t∂^2_{t_1}Y(t_1,t_2)+k(t_2)^2Y(t_1,t_2)=0\\
f_{tt}\partial_{t_1}Y(t_1,t_2)+2\epsilon f_t \partial^2_{t_1t_2}Y(t_1,t_2)+\epsilon^2\partial^2_{t_2}Y(t_1,t_2)=0
$$
The other degree of freedom is used up by setting $f_t(t,ϵ)=k(ϵt)$, so that with $K'=k$ you get $t_1=f(t,ϵ)=K(ϵt)/ϵ$ and thus
$$
Y(t_1,t_2)=c_1(t_2)\cos(t_1)+c_2(t_2)\sin(t_1)
$$
Then the second equations gives in the coefficients of $\cos(t_1)$ and $\sin(t_1)$ the conditions
$$
k'(t_2)c_2(t_2)+2k(t_2)c_2'(t_2)+ϵc_1''(t_2)=0
\\
k'(t_2)c_1(t_2)+2k(t_2)c_1'(t_2)-ϵc_2''(t_2)=0
$$
so that in first order $c_1(t_2)=\frac{d_1}{\sqrt{k(t_2)}}$ and $c_2(t_2)=\frac{d_2}{\sqrt{k(t_2)}}$.
Breaking the circle
Obviously, it is wrong to set $A=B=0$ as the method would prescribe. Meaning that you made an error in setting up the method, see below. Let's first look what happens when applying the method of undetermined coefficients
$$
x_1=Cτ\cosτ+Dτ\sinτ+\text{terms of double frequency}
$$
that is, you get resonance terms but no new contribution to the original homogeneous solution. Solving this gives $C=\frac{Bδ}2$, $D=-\frac{Aδ}2$ so that in total
$$
x_1=\frac{Bδ}2τ\cosτ-\frac{Aδ}2τ\sinτ+\frac{A}6(\cos2τ-3)+\frac{B}6\sin2τ
$$
On finite time spans these terms in $x_0+ϵx_1$ can be combined per Taylor to give an approximation using the sine and cosine of $(1+\frac{δϵ}2)τ$. This implies that the slow time should have been chosen as $T=δϵt$ or simply $T=ϵt$. Which does not lead toward the claim.
On solving the problem
You want to explore $a$ of the form $1+cϵ^2$. You can not do this by setting $δ=cϵ$, as all other quantities of the perturbation analysis are designed to be constant relative to $ϵ$. Or if you do it, then $δ$ is a constant and $1+δϵ$ is a linear, not a quadratic perturbation of $1$. By the claim of the task, you will not find the region of divergent resonance with this linear approach.
The perturbation system for the quadratic approach is
\begin{align}
\partial_{\tau\tau}x_0+x_0&=0, \tag{1}
\\
\partial_{\tau\tau}x_1+x_1&=-x_0\cos t, \tag{2}
\\
\partial_{\tau\tau}x_2+x_2&=-2\partial_{T\tau}x_0-cx_0-x_1 \cos t, \tag3
\end{align}
New version using standard trigonometric parametrization
The solution of the base equation has, as you also used, the form
\begin{align}
x_0(τ,T) &= A(T)\cos(τ)+B(T)\sin(τ)
\end{align}
Then the next order term satisfies
\begin{align}
∂_{ττ}x_1+x_1&=-\frac{A(T)}2(1+\cos2τ)-\frac{B(T)}2\sin2τ\\
x_1(τ,T) &= -\frac{A(T)}2 + \frac{A(T)}6\cos2τ + \frac{B(T)}6\sin2τ
\end{align}
Note that the frequency one components of the homogeneous/complementary solution were left out, as they would only replicate some fraction of the base solution.
In the next perturbation term we encounter the first $T$ derivative and the frequency perturbation, as well as the twice shifted base frequency, all sources for resonance.
\begin{align}
∂_{ττ}x_2+x_2&=-2∂_{Tτ}x_0 - cx_0 - \cosτ\,x_1 \\
&= 2A'(T)\sinτ - 2B'(T)\cosτ -cA(T)\cosτ - cB(T)\sinτ\\&\qquad
+ \frac{A(T)}2\cosτ - \frac{A(T)}{12} (\cos3τ+\cosτ) - \frac{B(T)}{12}(\sin3τ + \sinτ)
\end{align}
To avoid resonance, the frequency one terms have to cancel,
\begin{align}
2A'(T)&=cB(T) + \frac{B(T)}{12}&&=\left(\frac1{12}+c\right)B(T)
\\
2B'(T)&=-cA(T) + \frac{A(T)}2 - \frac{A(T)}{12}&&=\left(\frac{5}{12}-c\right)A(T)
\end{align}
This has a pair of real eigenvalues if $\left(\frac1{12}+c\right)\left(\frac{5}{12}-c\right)>0$, that is, for $-\frac1{12}<c<\frac{5}{12}$. As one of them is positive, this gives an exponentially growing term in the solution, leading to divergence as per the claim.
Old version using phase parametrization
An equivalent form of the solution of the base equation is
\begin{align}
x_0(τ,T)&=A(T)\cos(τ+B(T)), \\
x_0(τ,T)\cos(τ)&=\frac{A(T)}2[\cos(2τ+B(T))+\cos(B(T))],
\end{align}
Then the first perturbation term is computed as
\begin{align}
x_1(τ,T)&=\frac{A(T)}6\cos(2τ+B(T))-\frac{A(T)}2\cos(B(T)),
\\
x_1(τ,T)\cos(τ)&=\frac{A(T)}{12}[\cos(3τ+B(T))+\cos(τ+B(T))]\\
&~~~~-\frac{A(T)}2[\cos(τ+B(T))\cos^2(B(T))+\sin(τ+B(T))\sin(B(T))\cos(B(T))]
\\[.5em]
&=\frac{A(T)}{12}\cos(3τ+B(T))-\frac{A(T)}{12}[2+3\cos(2B(T))]\cos(τ+B(T))
\\&~~~~-\frac{A(T)}4\sin(2B(T))\sin(τ+B(T)),
\end{align}
Note that the frequency one components of the homogeneous/complementary solution were left out, as they would only replicate some variant of the base solution.
In the equation for the next term we encounter resonance which has to be resolved using the dependence of the parameters $A(T)$ and $B(T)$ on $T$.
\begin{align}
∂_{ττ}x_2+x_2&=2\Bigl[A'(T)\sin(τ+B(T))+A(T)B'(T)\cos(τ+B(T))\Bigr]\\
&~~~~-cA(T)\cos(τ+B(T))-x_1(τ,T)\cos(τ)
\\[.5em]
&=\Bigl[2A'(T)+\frac{A(T)}4\sin(2B(T))\Bigr]\sin(τ+B(T))\\
&~~~~+\Bigl[2A(T)B'(T)-cA(T)+\frac{A(T)}{12}[2+3\cos(2B(T))]\Bigr]\cos(τ+B(T))\\
&~~~~-\frac{A(T)}{12}\cos(3τ+B(T))
\end{align}
To remove the explicit growth in the form of polynomial coefficients we need to set the coefficients of the first two resonance terms, that is, of $\cos(τ+B(T))$ and $\sin(τ+B(T))$, to zero, giving the system
\begin{align}
8A'(T)&=-A(T)\sin(2B(T))\\
8B'(T)&=4c-\frac23-\cos(2B(T))
\\[1em]
\implies A(T)&=\frac{C}{4c-\frac23-\cos(2B(T))}
\end{align}
Now as long as $4c-\frac23\in(-1,1)\iff c\in(-\frac1{12},\frac{5}{12})$, the second equation has an infinite amount of simple equilibria, keeping $B(T)$ bounded between two of them. Asymptotically $B(T)$ will converge to the next stable equilibrium, that is, where $\sin(2B(T))$ is negative. But then the first equation will have a positive growth factor, so that the amplitude $A(T)$ is exponentially growing.
Outside that range, $A(T)$ is a periodic function of $B(T)$, and thus bounded.
This second approach is visibly more complicated due to multiple different applications of trigonometric identities, than the first one, and much harder to check for errors.
Best Answer
I don't believe it is a typo. The reason it is not secular is because it does not resonate with the frequency of the adjoint solution.
Formally, you have a linear operator $L$ applied to the solution $x_1$ with a RHS we can call $r_1$. So $L x_1 = r_1$ for this to have a solution, recall the condition from linear algebra that says if $Ax = b$ has a solution then $A^T y = 0$ and $y^T r = 0$. This is the solvability condition.
In the context here $L = \partial_{tt} + I$, is a self-adjoint operator and the solution $y = \sin, \cos$. The solvability condition is $\langle \sin t, \sin n t \rangle$ which is 0 unless $n = 1$.
EDIT: The solution of $Lu = 0$ are sin and cos. Secular terms cause blow-up and in this context occurs because the frequencies are the same. If you are unsure. Try solving $u'' + u = \sin 3t$ vs. $u'' + u = \sin t$ and see what happens as $t \to \infty$ for both.