What you should get an approximation for is depicted in the following plot of numerical solutions for some perturbation parameters.
This picture corresponds to some extended understanding of "boundary layer", as there is indeed some rapid compensation for the gap between the value $1$ at $t=0$ of the outer solution $y_{\rm outer}(t)=\cos(t)$ and the initial condition $y(0)=0$. This first segment has indeed also a width of $\sim\sqrt{ϵ}$, but then overshoots the outer solution and oscillates with frequency $1/\sqrt{ϵ}$, dampened by the friction term containing the first derivative giving a uniform hull curve for the amplitude of the oscillation of $e^{-t/2}$. Taking these building blocks to first satisfy $y(0)=0$ and then also $y'(0)=0$ gives the first order approximation
$$
y(t)\approx \cos(t)-e^{-t/2}\left(\cos\left(\frac{t}{\sqrt{ϵ}}\right)+\frac{\sqrt{ϵ}}2\sin\left(\frac{t}{\sqrt{ϵ}}\right)\right)
$$
Let's be a bit more ingeniuous about it. We know that largely $y(t)=\cos(t)+u(t)$ where $u$ will be small in later times. But then
$$
ϵu''+ϵu'+u=ϵ\cos(t)+ϵ\sin(t)
$$
Adjust the decomposition of $y$ for the new right side, now $y(t)=(1+ϵ)\cos(t)+ϵ\sin(t)+u(t)$, then the differential equation reads
$$
\begin{split}
ϵ[-(1+ϵ)\cos(t)-ϵ\sin(t)+u''(t)]&+ϵ[-(1+ϵ)\sin(t)+ϵ\cos(t)+u(t)]\\&+[(1+ϵ)\cos(t)+ϵ\sin(t)+u(t)]=\cos(t)
\end{split}
\\\iff\\
ϵu''+ϵu'+u=2ϵ^2\sin(t)\iff u''+u'+\frac1ϵu=2ϵ\sin(t)
$$
which now is essentially a homogeneous equation.
Approach via parametrized sinusoid
As this is a harmonic oscillator with damping, use the form proposed in the hint at the end of the section, set $u(t)=A(t)\cos(ωt+θ(t))$ with $ω=\sqrt{\frac1ϵ}$ and $A(t),θ(t)$ "slow" moving, in the sense that their derivatives are small against $ω$. Then
\begin{align}
u'(t)&=A'(t)\cos(ωt+θ(t))-A(t)\sin(ωt+θ(t))(ω+θ'(t)) \\
u''(t)&=A''(t)\cos(ωt+θ(t))-2A'(t)\sin(ωt+θ(t))(ω+θ'(t)) \\
& ~ ~ ~ ~- A(t)\cos(ωt+θ(t))(ω+θ'(t))^2-A(t)\sin(ωt+θ(t))θ''(t) \\
\end{align}
Comparing coefficients in $u''(t)+u'(t)+ω^2u(t)$ results in the identies
\begin{align}
0&=A''(t)+A'(t)- A(t)(2ω+θ'(t))θ'(t)\\
0&=-A(t)θ''(t)-2A'(t)(ω+θ'(t)) -A(t)(ω+θ'(t))\\[.7em]
\implies
2A(t)θ'(t)&=\sqrt{ϵ}[A''(t)+A'(t)-2A(t)θ'(t)^2]\\
2A'(t)+A(t)&=-\sqrt{ϵ}[A(t)θ''(t)+ (2A'(t)+A(t))θ'(t)]
\end{align}
so that in first approximation $A(t)=A_0e^{-t/2}$ and $θ(t)=θ_0$. Inserting into the right side gives the next order of approximation as
\begin{align}
2θ'(t)&=\sqrt{ϵ}[\tfrac14-\tfrac12]&&\implies& θ(t)=θ_0-\frac{\sqrt{ϵ}}8t\\
2A'(t)+A(t)&=0&&\implies& \text{unchanged}
\end{align}
Now adjust the coefficients for the initial conditions in
\begin{align}
y(t)&=(1+ϵ)\cos(t)+ϵ\sin(t)+A_0e^{-t/2}\cos\left((ω-\tfrac18\sqrt{ϵ})t+θ_0\right)\\
y'(t)&=-(1+ϵ)\sin(t)+ϵ\cos(t)-\tfrac12 A_0e^{-t/2}\cos\left((ω-\tfrac18\sqrt{ϵ})t+θ_0\right)
\\&~~~~-(ω-\tfrac18\sqrt{ϵ})A_0e^{-t/2}\sin\left((ω-\tfrac18\sqrt{ϵ})t+θ_0\right)
\\\hline
0=y(0)&=(1+ϵ)+A_0\cos\left(θ_0\right)\\
0=y'(0)&=ϵ-\tfrac12 A_0\cos\left(θ_0\right)-(ω-\tfrac18\sqrt{ϵ})A_0\sin\left(θ_0\right)\\
&=\tfrac12(1+3ϵ)-ω(1-\tfrac18ϵ)A_0\sin\left(θ_0\right)
\end{align}
It remains to solve the polar decomposition $A_0(\cos(θ_0),\sin(θ_0))=(-(1+ϵ), \tfrac12\sqrtϵ+O(ϵ^{3/2}))$ giving $θ_0=\pi-\tfrac12\sqrtϵ$ and $A_0=1+\frac98ϵ$.
Approach via WKB approximation
This all follows much more easily if the basis functions are computed via a WKB expansion $u(t)=e^{S/\delta}$, allowing for complex exponents, and then the real combinations are taken. The equation $ϵu''+ϵu'+u=0$ then leads to
$$
ϵ(δS''+S'^2)+ϵδS'+δ^2=0
$$
Also here one finds balancing for $δ^2=ϵ$ so that
$$
S'^2+1=-δ(S''+S')
$$
Using an expansion in powers of $δ$ (and setting $s_k=S_k'$) gives $s_0=\pm i$, $2s_0s_1=-s_0\implies s_1=-\frac12$, $s_1^2+2s_0s_2=-s_1\implies s_2=\mp\frac{i}8$
so that the basis solutions up to that approximation order are in their real form
$$
y_1(t)=e^{-t/2}\cos\left(\left(\frac1δ-\fracδ8\right)t\right),~~
y_2(t)=e^{-t/2}\sin\left(\left(\frac1δ-\fracδ8\right)t\right).
$$
Now use shortly $ω=\left(\frac1δ-\fracδ8\right)=\sqrt{\frac1ϵ}(1-\fracϵ8)$ and match the initial conditions for
\begin{align}
y(t)&=(1+ϵ)\cos(t)+ϵ\sin(t)+e^{-t/2}[A_0\cos(ωt)+B_0\sin(ωt)]\\
y'(t)&=-(1+ϵ)\sin(t)+ϵ\cos(t)-\tfrac12 e^{-t/2}[A_0\cos(ωt)+B_0\sin(ωt)]-ωe^{-t/2}[A_0\sin(ωt)-B_0\cos(ωt)]
\\\hline
0=y(0)&=(1+ϵ)+A_0\\
0=y'(0)&=ϵ-\tfrac12 A_0+ωB_0=\tfrac12(1+3ϵ)+ωB_0
\end{align}
so that more directly $A_0=-(1+ϵ)$ and $B_0=-\tfrac12\sqrtϵ(1+O(ϵ))$.
With some slightly more computation one can get also the next terms in the expansions of $A_0,B_0$
Breaking the circle
Obviously, it is wrong to set $A=B=0$ as the method would prescribe. Meaning that you made an error in setting up the method, see below. Let's first look what happens when applying the method of undetermined coefficients
$$
x_1=Cτ\cosτ+Dτ\sinτ+\text{terms of double frequency}
$$
that is, you get resonance terms but no new contribution to the original homogeneous solution. Solving this gives $C=\frac{Bδ}2$, $D=-\frac{Aδ}2$ so that in total
$$
x_1=\frac{Bδ}2τ\cosτ-\frac{Aδ}2τ\sinτ+\frac{A}6(\cos2τ-3)+\frac{B}6\sin2τ
$$
On finite time spans these terms in $x_0+ϵx_1$ can be combined per Taylor to give an approximation using the sine and cosine of $(1+\frac{δϵ}2)τ$. This implies that the slow time should have been chosen as $T=δϵt$ or simply $T=ϵt$. Which does not lead toward the claim.
On solving the problem
You want to explore $a$ of the form $1+cϵ^2$. You can not do this by setting $δ=cϵ$, as all other quantities of the perturbation analysis are designed to be constant relative to $ϵ$. Or if you do it, then $δ$ is a constant and $1+δϵ$ is a linear, not a quadratic perturbation of $1$. By the claim of the task, you will not find the region of divergent resonance with this linear approach.
The perturbation system for the quadratic approach is
\begin{align}
\partial_{\tau\tau}x_0+x_0&=0, \tag{1}
\\
\partial_{\tau\tau}x_1+x_1&=-x_0\cos t, \tag{2}
\\
\partial_{\tau\tau}x_2+x_2&=-2\partial_{T\tau}x_0-cx_0-x_1 \cos t, \tag3
\end{align}
New version using standard trigonometric parametrization
The solution of the base equation has, as you also used, the form
\begin{align}
x_0(τ,T) &= A(T)\cos(τ)+B(T)\sin(τ)
\end{align}
Then the next order term satisfies
\begin{align}
∂_{ττ}x_1+x_1&=-\frac{A(T)}2(1+\cos2τ)-\frac{B(T)}2\sin2τ\\
x_1(τ,T) &= -\frac{A(T)}2 + \frac{A(T)}6\cos2τ + \frac{B(T)}6\sin2τ
\end{align}
Note that the frequency one components of the homogeneous/complementary solution were left out, as they would only replicate some fraction of the base solution.
In the next perturbation term we encounter the first $T$ derivative and the frequency perturbation, as well as the twice shifted base frequency, all sources for resonance.
\begin{align}
∂_{ττ}x_2+x_2&=-2∂_{Tτ}x_0 - cx_0 - \cosτ\,x_1 \\
&= 2A'(T)\sinτ - 2B'(T)\cosτ -cA(T)\cosτ - cB(T)\sinτ\\&\qquad
+ \frac{A(T)}2\cosτ - \frac{A(T)}{12} (\cos3τ+\cosτ) - \frac{B(T)}{12}(\sin3τ + \sinτ)
\end{align}
To avoid resonance, the frequency one terms have to cancel,
\begin{align}
2A'(T)&=cB(T) + \frac{B(T)}{12}&&=\left(\frac1{12}+c\right)B(T)
\\
2B'(T)&=-cA(T) + \frac{A(T)}2 - \frac{A(T)}{12}&&=\left(\frac{5}{12}-c\right)A(T)
\end{align}
This has a pair of real eigenvalues if $\left(\frac1{12}+c\right)\left(\frac{5}{12}-c\right)>0$, that is, for $-\frac1{12}<c<\frac{5}{12}$. As one of them is positive, this gives an exponentially growing term in the solution, leading to divergence as per the claim.
Old version using phase parametrization
An equivalent form of the solution of the base equation is
\begin{align}
x_0(τ,T)&=A(T)\cos(τ+B(T)), \\
x_0(τ,T)\cos(τ)&=\frac{A(T)}2[\cos(2τ+B(T))+\cos(B(T))],
\end{align}
Then the first perturbation term is computed as
\begin{align}
x_1(τ,T)&=\frac{A(T)}6\cos(2τ+B(T))-\frac{A(T)}2\cos(B(T)),
\\
x_1(τ,T)\cos(τ)&=\frac{A(T)}{12}[\cos(3τ+B(T))+\cos(τ+B(T))]\\
&~~~~-\frac{A(T)}2[\cos(τ+B(T))\cos^2(B(T))+\sin(τ+B(T))\sin(B(T))\cos(B(T))]
\\[.5em]
&=\frac{A(T)}{12}\cos(3τ+B(T))-\frac{A(T)}{12}[2+3\cos(2B(T))]\cos(τ+B(T))
\\&~~~~-\frac{A(T)}4\sin(2B(T))\sin(τ+B(T)),
\end{align}
Note that the frequency one components of the homogeneous/complementary solution were left out, as they would only replicate some variant of the base solution.
In the equation for the next term we encounter resonance which has to be resolved using the dependence of the parameters $A(T)$ and $B(T)$ on $T$.
\begin{align}
∂_{ττ}x_2+x_2&=2\Bigl[A'(T)\sin(τ+B(T))+A(T)B'(T)\cos(τ+B(T))\Bigr]\\
&~~~~-cA(T)\cos(τ+B(T))-x_1(τ,T)\cos(τ)
\\[.5em]
&=\Bigl[2A'(T)+\frac{A(T)}4\sin(2B(T))\Bigr]\sin(τ+B(T))\\
&~~~~+\Bigl[2A(T)B'(T)-cA(T)+\frac{A(T)}{12}[2+3\cos(2B(T))]\Bigr]\cos(τ+B(T))\\
&~~~~-\frac{A(T)}{12}\cos(3τ+B(T))
\end{align}
To remove the explicit growth in the form of polynomial coefficients we need to set the coefficients of the first two resonance terms, that is, of $\cos(τ+B(T))$ and $\sin(τ+B(T))$, to zero, giving the system
\begin{align}
8A'(T)&=-A(T)\sin(2B(T))\\
8B'(T)&=4c-\frac23-\cos(2B(T))
\\[1em]
\implies A(T)&=\frac{C}{4c-\frac23-\cos(2B(T))}
\end{align}
Now as long as $4c-\frac23\in(-1,1)\iff c\in(-\frac1{12},\frac{5}{12})$, the second equation has an infinite amount of simple equilibria, keeping $B(T)$ bounded between two of them. Asymptotically $B(T)$ will converge to the next stable equilibrium, that is, where $\sin(2B(T))$ is negative. But then the first equation will have a positive growth factor, so that the amplitude $A(T)$ is exponentially growing.
Outside that range, $A(T)$ is a periodic function of $B(T)$, and thus bounded.
This second approach is visibly more complicated due to multiple different applications of trigonometric identities, than the first one, and much harder to check for errors.
Best Answer
I'm not sure if I would call this "balancing", but in introducing an auxiliary function $f$ you got a degree of freedom in the collection of functions $k,y,f$ that can be fixed to any functional relation. Additionally you gain a degree of freedom by splitting the time scales. So selecting to set the sum of these two terms to zero fixes, removes one degree of freedom and leaves you with two new equations $$ f^2_t∂^2_{t_1}Y(t_1,t_2)+k(t_2)^2Y(t_1,t_2)=0\\ f_{tt}\partial_{t_1}Y(t_1,t_2)+2\epsilon f_t \partial^2_{t_1t_2}Y(t_1,t_2)+\epsilon^2\partial^2_{t_2}Y(t_1,t_2)=0 $$ The other degree of freedom is used up by setting $f_t(t,ϵ)=k(ϵt)$, so that with $K'=k$ you get $t_1=f(t,ϵ)=K(ϵt)/ϵ$ and thus $$ Y(t_1,t_2)=c_1(t_2)\cos(t_1)+c_2(t_2)\sin(t_1) $$ Then the second equations gives in the coefficients of $\cos(t_1)$ and $\sin(t_1)$ the conditions $$ k'(t_2)c_2(t_2)+2k(t_2)c_2'(t_2)+ϵc_1''(t_2)=0 \\ k'(t_2)c_1(t_2)+2k(t_2)c_1'(t_2)-ϵc_2''(t_2)=0 $$ so that in first order $c_1(t_2)=\frac{d_1}{\sqrt{k(t_2)}}$ and $c_2(t_2)=\frac{d_2}{\sqrt{k(t_2)}}$.