I'll give an explicit expression for a family of solutions to your first problem (the more general one without the symmetry constraint).
Let us use Robert Israel's suggestion as in your last edit, and start from
\begin{equation}
XX^{\top}+\frac{1}{2}ABX^{\top}+\frac{1}{2}XB^{\top}A-A=0 \tag{4}
\end{equation}
You can factor it as
\begin{equation}
(X+\frac12 AB)(X^\top +\frac12 B^\top A) = A + \frac14 AB B^\top A.
\tag{*}
\end{equation}
The matrix $C = A + \frac14 AB B^\top A$ is positive semidefinite, hence it can be factored as $C=LL^\top$ (you can take $L=C^{1/2}$, for instance, or compute a Cholesky-like factorization where you stop the algorithm if you encounter a zero submatrix).
Then, a family of solutions is $X = LQ - \frac12 AB$, where $Q$ is any orthogonal matrix.
If $C$ is invertible, then this is the full set of solutions, since you can rewrite the equation as $L^{-1}(X+\frac12 AB)(X^\top +\frac12 B^\top A)L^{-\top} = I$, and hence the equality holds iff $Q=L^{-1}(X+\frac12 AB)$ is orthogonal. If $C$ is singular, I suspect there are more, and probably it is possible to find them all with some more work (change basis so that $C = \begin{bmatrix}C_{11} & 0 \\ 0 & 0\end{bmatrix}$).
But in any case, at this point the question is: do these solutions solve your problem, or do you have more constraints on $X$?
Remark: the condition $A \succeq 0$ can be relaxed: solutions exist if and only if $C\succeq 0$ (the LHS of $(*)$ is always positive semidefinite so $C$ must be too.)
EDIT: as noted by @loupblanc in his/her answer, this solution does not guarantee that $ABX^\top$ is symmetric for all choices of $Q$, hence it solves (4) but not in general (1). The missing condition is that $XB^\top A = (LQ-\frac12AB)B^\top A$ is symmetric, which is equivalent to requiring $LQB^\top A$ symmetric.
In the case in which $L$ is invertible, this is equivalent to requiring $QB^\top A L^{-\top}$ to be symmetric. A valid choice for $Q$ is given by computing the polar decomposition $L^{-1}AB=ZS$, with $Z$ orthogonal and $S$ symmetric, and taking $Q=Z$.
In the general case, of course there is an even more arcane matrix decomposition. We take a generalized SVD of the pair $(L^\top,B^\top A)$, i.e., $L^\top = U\Sigma_1Y, \, B^\top A = V\Sigma_2 Y$, where $Y$ is invertible, $\Sigma_1,\Sigma_2$ are diagonal with $\Sigma_1^2+\Sigma_2^2=I$, and $U,V$ are orthogonal (all square). Then, direct verification shows that choosing $Q=UV^\top$ gives a symmetric $XB^\top A$.
So in the end a solution to (1) exists iff $A + \frac14 ABB^\top A$ is positive semidefinite, and we can compute it in $O(n^3)$ using some lesser-known matrix decompositions (that are implemented stably, for instance, in Matlab or Scipy/Numpy).
Best Answer
This equation always has a solution: $X = O$. I'll assume throughout this answer that you're interested in a non-zero solution.
The equation $AX = XB$ is equivalent to $(A \otimes I - I \otimes B^T)\mathbf{x} = \mathbf{0}$, where $\otimes$ denotes the Kronecker product and $\mathbf{x}$ is the vectorization of $X$. Your question is thus equivalent to asking when the matrix $A \otimes I - I \otimes B^T$ is not invertible (i.e., when $0$ is not an eigenvalue of $A \otimes I - I \otimes B^T$).
Since the eigenvalues of $A \otimes I - I \otimes B^T$ are exactly the sums of the eigenvalues of $A$ and $-B$, the condition that you wrote ($\sigma_p(A) \cap \sigma_p(B) \neq \varnothing$) is in fact both necessary and sufficient.