Accuracy vs F-measure
First of all, when you use a metric you should know how to game it. Accuracy measures the ratio of correctly classified instances across all classes. That means, that if one class occurs more often than another, then the resulting accuracy is clearly dominated by the accuracy of the dominating class. In your case if one constructs a Model M which just predicts "neutral" for every instance, the resulting accuracy will be
$acc=\frac{neutral}{(neutral + positive + negative)}=0.9188$
Good, but useless.
So the addition of features clearly improved the power of NB to differentiate the classes, but by predicting "positive" and "negative" one missclassifies neutrals and hence the accuracy goes down (roughly spoken). This behavior is independent of NB.
More or less Features ?
In general it is not better to use more features, but to use the right features. More features is better insofar that a feature selection algorithm has more choices to find the optimal subset (I suggest to explore: feature-selection of crossvalidated). When it comes to NB, a fast and solid (but less than optimal) approach is to use InformationGain(Ratio) to sort the features in decreasing order and select the top k.
Again, this advice (except InformationGain) is independent of the classification algorithm.
EDIT 27.11.11
There has been a lot of confusion regarding bias and variance to select the correct number of features. I therefore recommend to read the first pages of this tutorial: Bias-Variance tradeoff. The key essence is:
- High Bias means, that the model is less than optimal, i.e. the test-error is high (underfitting, as Simone puts it)
- High Variance means, that the model is very sensitive to the sample used to build the model. That means, that the error highly depends on the training set used and hence the variance of the error (evaluated across different crossvalidation-folds) will extremely differ. (overfitting)
The learning-curves plotted do indeed indicate the Bias, since the error is plotted. However, what you cannot see is the Variance, since the confidence-interval of the error is not plotted at all.
Example: When performing a 3-fold Crossvalidation 6-times (yes, repetition with different data partitioning is recommended, Kohavi suggests 6 repetitions), you get 18 values. I now would expect that ...
- With a small number of features, the average error (bias) will be lower, however, the variance of the error (of the 18 values) will be higher.
- with a high number of features, the average error (bias) will be higher, but the variance of the error (of the 18 values) lower.
This behavior of the error/bias is exactly what we see in your plots. We cannot make a statement about the variance. That the curves are close to each other can be an indication that the test-set is big enough to show the same characteristics as the training set and hence that the measured error may be reliable, but this is (at least as far as I understood it) not sufficient to make a statement about the variance (of the error !).
When adding more and more training examples (keeping the size of test-set fixed), I would expect that the variance of both approaches (small and high number of features) decrease.
Oh, and do not forget to calculate the infogain for feature selection using only the data in the training sample ! One is tempted to use the complete data for feature selection and then perform data partitioning and apply the crossvalidation, but this will lead to overfitting. I do not know what you did, this is just a warning one should never forget.
The problem is small enough you can work it out by hand. For your example you have
$$
\begin{align*}
P(outlook = sunny| play=yes) &= \frac{2}{9}\\
P(temp = cool| play=yes) &= \frac{3}{9}\\
P(humidity=high| play=yes) &= \frac{3}{9}\\
P(windy=true| play=yes) &= \frac{3}{9}\\
P(play=yes) &= \frac{9}{14}.\\
\end{align*}
$$
Putting it all together you have
$$
\begin{align*}
P(play=yes|sunny, cool, high, true) &\varpropto \frac{2}{9} \left(\frac{3}{9}\right)^3 \frac{9}{4}\\
&\approx 0.0053,
\end{align*}
$$
which agrees with Mitchell. I don't use R, so I can't speak as to why the output is different. Obviously the package you're using is normalizing, but this shouldn't change the classification. If I had to guess I'd say it is the cross validation.
Best Answer
You're looking for reasons why your precision and recall are low, but your accuracy doesn't look that great either. A classifier that labels every example a zero would have an accuracy of 80%, which is close to what yours is.
I've built very few models in my professional life that were as predictive as I wanted them to be. There's a lot left unstated in the problem statement, but I'd venture that there is uncertainty that no amount of parameter tweaking is going to solve. Finding new or better features may be of more help, but that can be hard or expensive work.
Otherwise, I think you should play with the threshold that allows you to trade off recall for precision (within a single fitted model). If you're okay with "leaving money on the table," see what happens when you adjust the threshold such that the recall is 5%. Perhaps your precision will shoot up to 75% and there will still be enough positive classified cases for your needs. Or perhaps the situation is reversed and you're okay with lots of false positives, whereas you'd want to do the opposite.