Hint : Friction opposes tendency to move. Tension is produced if the string is stretched $very$ slightly. So, increase friction to maximum and then tension will act if necessary.
Ironically, you are thinking absolutely right. Give yourself a cookie.
From part $a$, we know that the blocks will be at rest at all angles below that.
You are also right as at very small angles there is no need of tension and we can ignore it to solve for, again, an angle condition. You have done excellent work. Congrats.
Now we come to the middle angles. Oh... they drive you insane, don't they?
Let's start. We can start our analysis from 2 blocks, 1 will give a contradiction and other will give a result, but I will start with the one giving contradiction. This will help you.
All angles are in degrees :
$\theta=35 $
Lets start by analysing Block A (No racism intended)
Gravity is trying to pull it down : $5*10*\sin(35)N=28.67N$
Friction comes to the rescue(up) : $50*\cos(35)N=16.38N$ // read my hint to know why friction is put max here
As it is at rest, $T=12.29N$
Now Block B is also at rest,
Weight = $114.71N$
max f= $81.92N$
$16.38+114.71=12.29+f$
$f=118.8N$
OOPS, it exceed max value. So, Lets start by analysing Block B. (I love alliteration)
Gravity trying : $114.71N$
Friction comes to the rescue(up) : $81.92N$
You can take from here I guess. calculate tension. Note that you have to revise your calculation for tension again as reaction friction force will be provided by A. Better assume it $f$ from starting FBD of B.
This will yield the correct answer. Friction will be less than max value for upper block. In most cases, You should start analysing with heavier block(my experience). Hope your doubts are cleared.
There is tension in the spring. It it extended and hence there is tension! It is the centre of mass that falls with acceleration $g$ rather then each individual mass. So the equation $$mg-T=mg$$ is invalid. As the two masses fall they will oscillate (getting closer and further away) and the tension will cycle.
Let us call the distance fallen by mass $A$, $x_A$ and that fallen by mass B $x_B$ the equation of motion for each mass is given by:
$$m \ddot x_A=mg+T$$
$$m \ddot x_B=mg-T$$
$T$ is a function of $x_A$ and $x_B$, ($T=k(x_B-x_A-L)$ where $k$ is the spring constant, and $L$ is the natural length) and we cannot assume that $\ddot x_A=g$ or $\ddot x_B=g$. These sorts of equations are called coupled differential equations and can be solved a number of ways.
Best Answer
I think I finally figured out what you are asking. You would like to do an overall force balance on the system using vectors, and have it give you the same result as your scalar analysis. Your problem is that it seems there are only vertical forces on m2 and only horizontal forces on m1, so how can they all properly cancel.
For mass 1, I think we can agree that the normal force of the table top cancels with the weight of mass 1. So we will exclude this from our vector force balance. Our system will be the two masses and the rope. The key to doing this correctly is to recognize that the pulley exerts a net force on our system. From a force balance on the pulley, we find that the pulley axle exerts a force of $(T\mathbf{i_x}+T\mathbf{i_y})$ on the pulley, and this force is transmitted via the pulley to the rope. So the overall force balance on our system is: $$m_1a\mathbf{i_x}+m_2a(-\mathbf{i_y})=(T\mathbf{i_x}+T\mathbf{i_y})-m_2g\mathbf{i_y}$$ If we resolve this force balance into components, we obtain the two scalar force balances in our previous answers.