You're studying a vector space of functions, and writing each such function as a linear combination of the elements of some basis of that vector space. When we work in terms of $e^{\alpha x}\sin\beta x$ and $e^{\alpha x}\cos\beta x$ instead of $e^{(\alpha\pm i\beta)x}$, we change our choice of basis, so the coefficients change. You can think of it as rotating axes. As @JyrkiLahtonen notes, $c_1=i(A-B)$ and $c_2=A+B$. (Well, they had them the other way round, but you did choose $c_1$ as your $e^{\alpha x}\sin\beta x$ coefficient.)
To go from (2) to (3), use the compound angle formula $\sin(\beta x+\gamma)=\cos\gamma\sin\beta x+\sin\gamma\cos\beta x$, so you have to solve $c\cos\gamma=c_1,\,c\sin\gamma=c_2$ using $c^2=c_1^2+c_2^2$. I'll leave the rest to you.
So,
$${y'' + yy' = 0}$$
is equivalent to
$${\frac{d}{dx}[y'] + \frac{1}{2}\frac{d}{dx}[(y)^2]=0}$$
Rearranging and integrating:
$${\Rightarrow \frac{d}{dx}[y'] = -\frac{1}{2}\frac{d}{dx}[(y)^2]}$$
$${\Rightarrow y' = -\frac{1}{2}(y)^2 + c_1}$$
As you said. Although you got here by chain rule. You said
$${y'' = \frac{dy'}{dx} = \frac{dy'}{dy}\frac{dy}{dx}}$$
And so the ODE cancels to
$${\frac{dy'}{dy}\frac{dy}{dx} + y\frac{dy}{dx}=0}$$
And obviously cancelling the ${y'}$ (provided ${y'\neq 0}$) gives
$${\frac{dy'}{dy} + y = 0}$$
Then integrating both sides with respect to $y$, you get the same thing. Both are completely valid!
Anyways, this is just a separable ODE. Note
$${\Rightarrow \frac{dy}{dx} = c_1-\frac{1}{2}y^2}$$
$${\Rightarrow \frac{1}{c_1-\frac{1}{2}y^2}dy = dx}$$
EDIT: This is the stage that is incorrect if ${y'(0)=0}$ so ${c_1-\frac{1}{2}y^2(0)=0}$. You are dividing through by $0$ at this stage, so every step after this becomes invalid. When doing divisions like this of general expressions - you should technically always check to make sure you aren't dividing by a potential $0$. In this case - it only happens if ${y'(0)=0}$. Sorry - I should have stated this previously :)
Anyways, provided we haven't just divided by $0$:
$${\Rightarrow\int \frac{1}{c_1-\frac{1}{2}y^2}dy = x + c_2}$$
Now we are left to integrate
$${\int\frac{1}{c_1-\frac{1}{2}y^2}dy}$$
You should obviously firstly take the case ${c_1=0}$ (this will you give you a nice trivial solution, so I will leave this to you). Otherwise, if ${c_1>0}$, after factoring
$${=\int\frac{1}{c_1}\frac{1}{1-\frac{1}{2c_1}y^2}dy=\int\frac{1}{c_1}\frac{1}{1-\left(\frac{1}{\sqrt{2c_1}}y\right)^2}dy}$$
And to integrate this, you should do a substitution of ${u=\frac{1}{\sqrt{2c_1}}y}$ and use partial fractions. You then must take the case ${c_1<0}$. I won't write out the steps again, but in this case you cannot use the trick ${\sqrt{c_1}^2 = c_1}$ since the square root will be imaginary and things get complicated. The answer will be in terms of ${\tan}$ in this case.
The bottom line is - you really cannot ignore the constants, since the answer changes depending on what that constant is. Obviously for the initial conditions, you do not know the sign of ${c_1}$, so what do you do? Well, going back to
$${y' + \frac{1}{2}y^2 = c_1}$$
Plugging in ${x=0}$ will of course give you the value of ${c_1}$, and so you will know which anti-derivative to take based on this :)
Upon request, I could add more steps (for example, I could show how to continue to compute the anti-derivative and show the answer to the first part), but first - can you try taking it from here?
Best Answer
$$ye^{2x}=\int{e^{2x}\left(xe^{x}+c_1e^{x}\right)}dx$$ $$\implies ye^{2x}=\frac13xe^{3x}-\frac19e^{3x}+\frac13{c_1}e^{3x}+c_2\tag{5.1}$$ Let this equality $(5.1)$ can be written as $$ye^{2x}=\frac13xe^{3x}+{c_1}^{\prime}e^{3x}+c_2\tag{5.2}$$ Comparing equation $(5.1)$ and $(5.2)$ ,$~\color{red}{\text{(that means comparing the coefficients of similar powers of $e$ in equation $(5.1)$ and $(5.2)$)}}~$ we have
$$c_1~'=\frac13{c_1}-\frac19$$
${}$
${}$
Here the differential equation is $$y^{\prime\prime}+y^{\prime}-2y=e^x\tag1$$ $$\implies \left(D^2+D-2\right)y=e^x\qquad \text{where}\quad D\equiv \frac{d}{dx}$$
Taking the equation $$y^{\prime\prime}+y^{\prime}-2y=0\tag2$$ and putting $~y=e^{mx}~$ in it, then we have
$$m^2+m-2=0$$called auxiliary equation. Roots of the auxiliary equation is $~m=-2,~1~$.
Then the solution of $(2)$ is $$y~=~c_1~ e^x ~+~c_2~e^{-2x}\qquad \text{where}\quad c_1,~c_2~\text{are constants.}$$
This $$c_1~ e^x ~+~c_2~e^{-2x}$$ is called the complementary function (C.F.) of $(1)$.
For non-homogeneous part, we have to find the particular integral (P.I.)
P.I.$~=\frac{1}{D^2+D-2}~e^x=\frac{1}{3}~\left(\frac{1}{D-1}-\frac{1}{D+2}\right)~e^x=\frac{1}{3}~\left(x~e^x-\frac{1}{3}~e^x\right)$
Hence the general solution of $(1)$ is $$y(x)~=~C.F.~+~P.I.$$ $$\implies y(x)=~c_1~ e^x ~+~c_2~e^{-2x}~+~\frac{1}{3}~\left(x~e^x-\frac{1}{3}~e^x\right)$$ $$\implies y(x)=\frac13xe^{x}+{c_1}^{\prime}e^{x}+{c_2}e^{-2x}$$where $~{c_1}^{\prime}={c_1}-\frac19~~$ and $~ c_2~$ are constants.