The marginal density is given by
$$
f_X(x)=\int_{-\infty}^\infty f_{X,Y}(x,y)\,\mathrm dy,\quad x\in\mathbb{R}.
$$
Now, this equals
$$
\int_{0}^1 \pi x\cos\left(\frac{\pi y}{2}\right)\,\mathrm dy,\quad \text{if }\;0\leq x\leq 1
$$
and $0$ otherwise.
To solve this problem, I need to draw a picture. I strongly recommend that you do so also.
Note that since $0\lt x\lt 1$, we have $0\lt y\lt 2$.
So draw the rectangle with corners $(0,0)$, $(1,0)$, $(1,2)$, and $(0,2)$.
Draw the lines $y=x$ and $y=x+1$.
Our random variable lives in the rectangle, and between these two lines.
Now finding the (marginal) density function of $X$ is easy. We have to "integrate out" $y$. So $y$ will travel from $x$ to $x+1$.
In principle, finding the density function of $Y$ is also easy, we have to integrate out $x$.
But if you look at the picture, you can see that we will have to break up the integral into two parts.
If $0\lt y\le 1$, we will be integrating from $x=0$ to $x=y$. From $1\lt y\le 2$, we will be integrating from $x=y-1$ to $x=1$.
Best Answer
You would integrate over the support of X. The first bounds you gave cannot be correct because density functions must be nonnegative which is false for x=y=2. If the second bounds are correct then integrate from y to 1. Check if the resulting density integrates to one.