You should construct an "upper envelope diagram" that describes the best response pure strategies and payoffs of player 2 against the mixed strategies of player 1. For an example, see the left part of figure 1 in the following paper:
D. Avis, G. Rosenberg, R. Savani, and B. von Stengel (2010), Enumeration of Nash equilibria for two-player games. Economic Theory 42, 9-37.
http://www.maths.lse.ac.uk/Personal/stengel/ETissue/ARSvS.pdf
(That paper also includes a complete technical exposition of the general problem for m by n games.)
Here is an example of the upper envelope diagram for a 2x5 game, where player I has pure strategies A and B and player 2 has pure strategies a,..,e.
What you are interested in is the "vertices" of this upper envelope. The two vertices at the far left and right correspond to the best responses against the two pure strategies of player one (so for these vertices you just check if the corresponding strategy of player 1 is a best response to this best response of player 2).
Vertices between these correspond to mixed strategies of player 1 that make player 2 indifferent between (at least) two different pure strategies.
Suppose we are at such a vertex defined by the pure strategies of player 2 a and b. For this vertex you need to look at the two columns of player 1's payoff matrix that correspond to a and b and check if the best response of player 1 against a is different from the best response of player 1 against b (or is tied, but let's ignore that case for simplicity). If these two best responses to a and b differ then you can find a mixed strategy for player 2 that makes player 1 indifferent between his two pure strategies. Then this strategy of player 2, paired with the strategy of player 1 given by the point on the x-axis in the best response diagram, is a Nash equilibrium.
You need to check this for every vertex (for the example above, corresponding to {d} at the left, then {d,b},{b,a}, and {a,e} in the interior, and {e} on the far right). Checking the far left and far right vertices correspond to checking for pure Nash equilibria. Checking the "interior" vertices corresponds to looking for equilibria where both players use 2 pure strategies.
Things can be complicated by degeneracies in the payoffs, e.g., three lines meeting in one point on the upper envelope, but I won't go into that here.
To investigate this, you can use our software:
http://www.gametheoryexplorer.org/
Pure strategies can be seen as special cases of mixed strategies, in which some strategy is played with probability $1$. In a finite game, there is always at least one mixed strategy Nash equilibrium. This has been proven by John Nash[1].
There can be more than one mixed (or pure) strategy Nash equilibrium and in degenerate cases, it is possible that there are infinitely many. In a well-defined sense (open and dense in payoff-space), almost every finite game has a finite and odd number of mixed strategy Nash equilibria.
A typical example of a game with more than one equilibrium is Battle of Sexes, which has two pure strategy equilibria and one completely mixed equilibrium, meaning every strategy is played with positive probability.
[1]: J.Nash. Non-Cooperative Games. http://www.cs.upc.edu/~ia/nash51.pdf
Best Answer
Okay, let's try this. For the 3x2 game, there are two obvious pure strategy Nash equilibria, (8,8) and (6,6). Are there any mixed equilibria?
Start with player 2, the Column player, who chooses X with probability $r$ and Y with probability $(1-r)$, and player 1, the Row player, choose a with probability $p$, b with probability $q$, and c with probability $(1-p-q)$, so I've slightly changed OP's notation. Column's payoff is $$r[8p+2q+(1-p-q)5]+(1-r)[0p+6q+(1-p-q)5].$$ Taking the derivative with respect to Column's choice variable, $r$ yields $$8p-4q.$$ If we set that equal to zero, we get $p=1/2q$, but that is the wrong way to look at the problem. What the derivative really says is that if $p<1/2 q$ then $r=0$, because the derivative is negative and the maximum occurs at the lower endpoint, 0. Similarly, if $p>1/2 q$ then $r=1$ because the derivative is positive.
So the only hope for a mixed strategy equilibrium is if $p=1/2 q$. Will that work? Look at the problem from the viewpoint of player 1, who plays Row. Her expected value is $$pr8 +0 +qr2+q(1-r)6+(1-p-q)5.$$ Take the derivative with respect to $p$, simplify, and get $8r=5$, or $r=5/8$. As before, this says that if $r<5/8$ then $p=0$. Take the derivative with respect to $q$, and a similar approach shows that $r<1/4$ implies $q=1.$
So, the only places where $p$ and $q$ are not either zero or one, is $p=5/8$ and $q=1/4.$ But then we don't have $p=1/2q$ and so there is not mixed equilibrium. or so it seems to me.