As Ted Schifrin rightly noted, you discarded a lot of information by combining three equations into two the way you did. Instead, you could proceed by back-substitution: from the second equation, we have $x_3=-x_1+(1-\lambda)x_3$, which upon substitution into the other equations produces $$\begin{align} (3-\lambda)x_1+(3\lambda-5)x_2 &= 0 \\ (\lambda-3)x_1+(\lambda^2-6\lambda+7)x_2 &= 0. \end{align}$$ Adding these two equations leaves $(\lambda^2-3\lambda+2)x_2=0$, which gives us two of our eigenvalues, $1$ and $2$. You can get the last eigenvalue, $3$, “for free” by examining the trace of $A$, or grind it out by setting $x_2=0$ in your original equations, which eventually leads to $(\lambda-3)x_1=0$.
As far as your row-reduction goes, the smaller example suggests that you’re using an invalid inference to decide what the eigenvalues of the matrix are. All that you can really say about the smaller example is that the elementary row operation that you performed is invalid when $\lambda=1$. That, however, doesn’t imply that $\lambda$ is an eigenvalue of $B$. Instead, begin by swapping rows so that you don’t have to divide by a term that might vanish and go from there: $$\begin{bmatrix}1-\lambda&5\\3&3-\lambda\end{bmatrix} \to \begin{bmatrix}3&3-\lambda\\1-\lambda&5\end{bmatrix} \to \begin{bmatrix}3&3-\lambda\\0&\frac13(\lambda-6)(\lambda+2)\end{bmatrix}.$$ This is rank-deficient when $\lambda=6$ or $\lambda=-2$, which are indeed the eigenvalues of $B$.
Similarly for $A$, all that you can really conclude from examining the denominators of entries in the reduced matrix is that you have to try something else for the values at which they vanish. You can’t immediately conclude that those values are eigenvalues of the matrix. Avoiding dividing by expressions that involve $\lambda$ by judicious row swaps or simply by not normalizing the pivot to $1$, I end up with $$\begin{bmatrix}1&1&\frac12(5-\lambda)\\0&2-\lambda&\frac12(3-\lambda)\\0&0&-\frac12(\lambda-3)(\lambda-1)\end{bmatrix}.$$ We immediately get the eigenvalues $1$ and $3$ from this, and compute $2$ from the trace of $A$. Alternatively, since we have $2-\lambda$ in a pivot position, set $\lambda=2$ and examine the resulting matrix. It has two identical rows, so is also rank-deficient.
All that aside, I prefer to try Hagen von Eitzen’s answer to the question that you linked before doing anything else: namely, see if I can guess any eigenvectors by trying simple linear combinations of rows and columns. In this case, subtracting the last column from the first produces $(3,0,-3)^T$, so $(1,0,-1)^T$ is an eigenvector with eigenvalue $3$. Subtracting the second column from the first gives $(2,-2,0)^T$ for another eigenvalue. Another way to find this one is to add the first and last rows to each other (left and right eigenvalues of a matrix are identical). The last eigenvector can always be computed from the trace: $6-3-2=1$.
$P$ is given and the question asks you about the properties of $P_1$ and $Q$. You cannot permute the rows and columns of $P$ at will, although you may permute the first $r$ rows/columns and the last $n-r$ rows/columns separately, so that the original block structure remains intact.
Part (a) is easy. If $R\ne0$, then some entry $r_{ik}$ is positive. Therefore
$$
1=\sum_jr_{ij}+\sum_lq_{il}\ge r_{ik}+\sum_lq_{il}>\sum_lq_{il}.
$$
Part (b) as it stands is not true. Here is a counterexample:
$$
\left[\begin{array}{ccc}0&0&1\end{array}\right]
\left[\begin{array}{c|cc}1&0&0\\ \hline\frac12&\frac12&0\\ 0&0&1\end{array}\right]
=\left[\begin{array}{ccc}0&0&1\end{array}\right].
$$
However, the problem statement is true if every row of $R$ is nonzero. In this case, every row sum of $Q$ is less than $1$. Hence the induced maximum norm $\|Q\|_\infty$ is less than $1$ and $\lim_{n\to\infty}Q=0$.
Now suppose $v^TP=v^T$. Then $v^TP^2=(v^TP)P=v^TP$ and in turn, $v^TP^n=v^T$ for every $n\ge1$. Therefore $\lim_{n\to\infty}v^TP^n$ exists and is equal to $v^T$. However, if we partition $v^T$ as $(x^T,y^T)$, then $v^TP^n$ is in the form of $(\ast,y^TQ^n)$. It follows that
$$
v^T=\lim_{n\to\infty}v^TP^n=\lim_{n\to\infty}(\ast,y^TQ^n)=(\ast,\,y^T\lim_{n\to\infty}Q^n)=(\ast,0).
$$
Best Answer
Hint:
You are so close: simplifying by $x$,
$$\frac{\lambda -a}b=\frac b{\lambda -a}$$ can be solved for $\lambda$.