This question is kinda weird since you didn't say anything about repetition. Supposing our sequence of length n allows repetition, here is how I would approach it:
We want to create sequences of length n from a set of $k$ characters allowing repetition with the constraint that at most two adjacent characters can be the same.
Let's use the inclusion-exclusion principle.
First, the number of sequences of length $n$ from $k$ elements allowing repetition is: $k^n$.
Now let's define the following sets for $ 3 \leq p \leq n$:
$$
S_p=\{\text{Sequences of length n with p adjacent characters the same}\}
$$
Therefore our final answer is: $k^n- |\bigcup_{p=3}^n S_p|$. Let's then calculate the cardinality of that union.
Clearly those sets are all disjoint, therefore:
$$
|\bigcup_{p=3}^n S_p| = |S_3| + |S_4| + \cdots + |S_n|
$$
And we can use the following strategy to calculate the cardinality of $|S_p|$: From our $k$ characters, select $1$ and place it in $p$ adjacent positions. There are $n-p+1$ possibilities to choose adjacent positions. Now we need to let the remaining $n-p$ positions to select, without repetition, characters from the $k-1$ remaining, and that can be done in $(k-1)(k-2)\cdots (k-n+p)=\frac{(k-1)!}{(k-n+p-1)!}$ ways. Therefore, by multiplication principle we get:
$$
|S_p|=k\cdot (n-p+1) \cdot \frac{(k-1)!}{(k-n+p-1)!} = \frac{k!(n-p+1)}{(k-n+p-1)!}
$$
which follows that:
$$
|\bigcup_{p=3}^n S_p| = \sum_{p=3}^{n} \frac{k!(n-p+1)}{(k-n+p-1)!}
$$
Finally, our answer is:
$$
k^n - \sum_{p=3}^{n} \frac{k!(n-p+1)}{(k-n+p-1)!}
$$
Sanity Check: Suppose $n=4$ and $K=\{a,b\}$. Therefore all sequences are:
\begin{align*}
(a,a,a,a)&\\
(a,a,b,b)&,(a,b,a,b)\\
(a,a,a,b)&,(a,a,b,a),(a,b,a,a),(b,a,a,a)\\
(b,b,b,b)&\\
(b,b,a,a)&,(b,a,b,a)\\
(b,b,b,a)&,(b,b,a,b),(b,a,b,b),(a,b,b,b)\\
(b,a,a,b)&,(a,b,b,a)
\end{align*}
There are $2^4=16$ sequences. Removing the ones that we don't want to count, that is, sequences with three or more adjacent repetitions, we have the remaining sequences:
$$
(a,a,b,b),(a,b,a,b)\\
(a,a,b,a),(a,b,a,a)\\
(b,b,a,a),(b,a,b,a)\\
(b,b,a,b),(b,a,b,b)\\
(b,a,a,b),(a,b,b,a)
$$
There are $10$ sequences. Now let's check if our answer is going to validate to the same number:
\begin{align*}
k^n - \sum_{p=3}^{n} \frac{k!(n-p+1)}{(k-n+p-1)!} &= 2^4 - \frac{2!(4-3+1)}{(2-4+3-1)!} - \frac{2!(4-4+1)}{(2-4+4-1)!}\\
&= 16 - \frac{2!(2)}{(0)!} - \frac{2!(1)}{(1)!}\\
&= 16 - 4 - 2 = 10
\end{align*}
So it seems to make sense. I'm not sure if it's 100% correct, but that's what I've done to try to solve it. Hope it helps!
Best Answer
I think there's a misunderstanding in the problem statement. They're only considering an alphabet of size $K$, not 52. The factor arising from choosing the substitution from 52 allowed characters will, as you noted, be enormous. But they're not considering it.
I'll start by answering the question as stated:
Let $N(L, K)$ be the number of distinct strings of length $L$ made of given $K$ characters (or $K$ generic placeholder characters), with each character used at least once, for $L\geq K$.
$$N(L, K) = K^L - {K\choose 1}(K-1)^L + {K\choose 2}(K-2)^L\dots = \sum\limits_{i=0}^{K-1} (-1)^i {K \choose i}(K-i)^L$$ This uses the inclusion-exclusion principle. Check this answer: https://math.stackexchange.com/a/664790/52735
From $N(L, K)$, we can remove the similar strings by dividing by $K!$ (number of permutations of $K$ letters).
Each remaining string of length $L$ with $K$ generic characters can be turned into a real string by choosing an appropriate substitution from the 52 possible letters. Two such assignments will give similar strings.
Putting these together:
$$\mathrm{ExpectedAns}(L, K) = \frac{N(L, K)}{K!} \times \frac{^{52}\mathrm{P}_{K} \times (^{52}\mathrm{P}_K-1)}{2}$$ with $N(L, K)$ given above (for $L\geq K$). This clearly doesn't match $\mathrm{Ans}(5, 2) = 15$.
My guess is they're only allowing for $K$ given characters to be substituted. So maybe this is what they're looking for
$$\mathrm{Ans}(N, K) = \frac{N(L, K)}{K!} \times \frac{^{K}\mathrm{P}_{K} \times (^{K}\mathrm{P}_K-1)}{2}$$ $$\mathrm{Ans}(N, K) = \frac{N(L, K)}{K!} \times \frac{K! \times (K!-1)}{2}$$ For $(5, 2)$ this gives $15$. The factor $N(L, K)/K!$ is referred to as Stirling numbers of the second kind: https://en.wikipedia.org/wiki/Stirling_numbers_of_the_second_kind