Your first two theories, Φ in the spinor rep of SU(2), and φ in the vector rep of O(4), are dealt with correctly, with 3 generators broken in both cases, so 3 goldstons and one massive field.
You have completely messed up the counting and symmetry structure of your latter theory. The first form, with a complex triplet ξ, is SU(3)-, not just SU(2), invariant, and this SU(3) breaks down to the residual SU(2) by the v.e.v., so 8-3=5 broken generators, and thus 5 goldstons, and one residual massive field, just as in the language of your O(4)/O(3) vector representation model.
I am unclear as to how you concluded, erroneously, that "there is no residual SU(2)". There is: it mixes up the components not involving the v.e.v. So, for example, if the v.e.v. is dialed to the 3rd component, the SU(2) subgroup mixing up the upper two components ($\lambda_1, \lambda_2,\lambda_3$ Gell-Mann matrices) is unbroken. You ought to brush up on the standard elementary SSB counting arguments, which your teacher must have assigned to you, Ling-Fong Li (1974)
So you really have two closely related questions here. Let's start with your easier one. You wrote the Lagrangian
$\mathcal{L}(\phi_1,\phi_2)=\frac{1}{2}\partial_\mu \phi_1\partial^\mu \phi_1+\frac{1}{2}\partial_\mu\phi_2\partial^\mu \phi_2-\frac{3}{4}M^2(\phi_1^2+\phi_2^2)-\frac{1}{2}M^2\phi_1\phi_2$
And asked for the symmetry group. I can see that the first three terms have an $SO(2)$ symmetry, but this symmetry is broken by the $\phi_1 \phi_2$ cross-term. You have done a change of variables,
$\phi_{M^2}=\frac{\phi_2-\phi_1}{\sqrt{2}}, \qquad \phi_{2M^2}=\frac{\phi_1+\phi_2}{\sqrt{2}}$,
which makes this fact more obvious. The Lagrangian becomes
$\mathcal{L}_2(\phi_{M^2},\phi_{2M^2})=\frac{1}{2}\partial_\mu \phi_{M^2}\partial^\mu \phi_{M^2}+\frac{1}{2}\partial_\mu\phi_{2M^2}\partial^\mu \phi_{2M^2}-\frac{1}{2}[(2M^2)\phi_{2M^2}^2+M^2\phi_{2M^2}^2)]$
and you are correct that, if the diagonalized fields had equal masses, then there would be an $SO(2)$ symmetry. If you work backwards through your change of variables, you should find that the diagonal fields have equal masses if and only if the $\phi_1 \phi_2$ term had vanished in the first place.
Now let's look at your more general Lagrangian:
$\mathcal{L}_3(\phi_1,\dots,\phi_n)=\sum_{i,j=1}^n\left(\frac{1}{2}K_{ij}\partial_\mu\phi_i\partial^\mu\phi^j-M_{ij}\phi^i\phi^j\right)-V(\phi_i)$
Since $K_{ij}$ is real and symmetric, it can be diagonalized by an orthogonal matrix. Finally, by rescaling the fields, you can set $K_{ij} = \delta_{ij}$ to give yourself the standard kinetic term. This term on its own is $SO(n)$ invariant.
The question that remains is, after doing this field redefinition, what does $M_{ij}$ look like? Since the kinetic term is now $SO(n)$ invariant, and $M_{ij}$ is symmetric, we can also diagonalize $M_{ij}$. However, we cannot do any more rescalings, so the new Lagrangian, written in terms of mass eigenfields, will generically have fields with different masses. If all the masses are different, then all the $SO(n)$ symmetry is broken. If any masses are the same, then their multiplicities can leave you a residual $SO(k) \times SO(\ell) \times \ldots$, etc.
Edited to add: You also need to look at the symmetries of your potential $V(\phi)$, which may further break symmetry!
Best Answer
You don't need the first and second terms for invariance as such. But you should recognize that these are simply the kinetic and interaction terms for the axion field. Are you proposing to include a field without kinetic term into the Lagrangian?
You want an additional anomaly coming from the axion term here, so that the total anomaly in the Peccei-Quinn model cancels.
You "use" the Peccei-Quinn $\mathrm{U}_\text{PC}(1)$ symmetry when you spontaneously break it under the influence of the periodic effective potential generated by the anomalous currents. The resulting physical axion is the Goldstone boson of the $\mathrm{U}_\text{PC}(1)$ broken by the VEV taken by the axion field w.r.t. to this potential.