so we want a vector $(x,y,z)$ such that when premultiplied by A gives us b.
so therefore $x+y-2z=3$, $y+z=-1 $and $-2x-y-5z=-1$. We are allowed multiply these equations by constants and add these equations. (left side with left and right with right. As you can see, (if we ignore the variables and just look at the coefficients this is the same as adding one row of the matric to another and multiplying one row with another. Therefore we can work with matrices. Where each row contains the coefficients of the variables on the first three columns and the constant to which they add up on the right side.
So at the start we have.
$$\begin{bmatrix}1&1&-2&3\\0&1&1&-1\\-2&-1&5&-7\end{bmatrix}$$
Then we switch the second and third rows
$$\begin{bmatrix}1&1&-2&3\\-2&-1&5&-7\\0&1&1&-1\end{bmatrix}$$
add twice the first one to the second one
$$\begin{bmatrix}1&1&-2&3\\0&1&1&-1\\0&1&1&-1\end{bmatrix}$$
and then subtract the second one from the third one to get
$$\begin{bmatrix}1&1&-2&3\\0&1&1&-1\\0&0&0&0\end{bmatrix}\;.$$
We can simplify a bit more by subtracting the second row from the first:
$$\begin{bmatrix}1&0&-3&4\\0&1&1&-1\\0&0&0&0\end{bmatrix}\;.$$
This says that $x_1-3x_3=4$ and $x_2+x_3=-1$, or
$$\left\{\begin{align*}
x_1&=4+3x_3\\
x_2&=-1-x_3\;,
\end{align*}\right.$$
where $x_3$ can be any real number whatsoever. Taking $x_3=0$ makes the calculation very easy and gives us one solution:
$$x=\begin{bmatrix}4\\-1\\0\end{bmatrix}\;.$$
But this is clearly not unique: we get infinitely many other solutions by changing the value of $x_3$.
I think you have the right idea for the first part. In particular, the image of the unit vector $e_1$ is given by
$$
T(e_1) = \pmatrix{-1&1\\1&1} \pmatrix{1\\0} = \pmatrix{-1\\1}
$$
so, the image of $e_1$ is simply $(-1,1)$.
It's clear that you're confused about the second part. The "unit square" here means the set of all vectors (not just unit vectors) whose components are the coordinates of some point in the unit square. That is, we want every vector $(x,y)$ for which $0 \leq x \leq 1$ and $0 \leq y \leq 1$.
In practice, however, it is not necessary to consider every vector in this space. The real approach here is to note that the image of your quadrilateral will be another quadrilateral, and note that each corner of the unit square will be mapped to a corner of the image.
I cannot say whether a sketch will be required. However, I will say that a sketch will be helpful enough that you should make one even if it isn't.
As for the last part: you should think of the transformation as a reflection followed by a scaling.
Best Answer
Hint: Notice that: $$ \vec b = x\vec e_1 + y\vec e_2 $$ so by the linearity of the given transformation $T$, we know that: $$ T(\vec b) = xT(\vec e_1) + yT(\vec e_2) = x\vec y_1 + y \vec y_2 $$