I will use established notations, when initial TF-IDF matrix stores documents at columns, and rows correspond to term occurrences or term's tf/idfs.
Let $A$ be $(n \times d)$ matrix with $d$ domuments and $n$ dimensions.
The standart method to do feature reduction in text mining is latent semantic indexing. The key idea is applying a little modification of SVD decomposition for $n \times d$ TF-IDF matrix (or just word occurrence matrix).
Particularly, let our initial matrix $A$ be decomposed:
$$A = S\times D \times T^t,$$ where $S, D, T^t$ have dimensions $(n \times r), (r \times r), (r \times d)$ respectively and $D$ is a diagonal matrix with $A$'th sorted singular values on diagonal: $D = diag(d_1,d_2,\dots,d_r), d_1 \le \dots d_r$.
The modification performed in latent semantid indexing is truncating the matrix $D$ so that only $k \le r$ largest singular values remained. It can be shown, that
$A \approx A_k = S_k \times D_k \times T_k^t$, where $S_k$ is $n \times k$ matrix of first $k$ columns of $S$, $T_k^t$ is $k \times d$ matrix of first $k$ rows of $T^t$, and $D_k = diag(r_1,\dots,r_k)$. The matrix $T_k$ is a concept document matrix, which rows store reduced description of document. Further you apply text-mining algorithms to that matrix as you could apply them to initial TF-IDF matrix.
Have you thought about storing the term frequency and the document frequency separately for the trained set, then when you add a new document you can update the document frequency of the new trained set, i.e. including the new document, and calculate tf-idf. Or am I missing something?
Best Answer
Create a simple Naive Bayes classifier over a corpus of positive and negative sentiments , and weights words with probability of it is positive or not , for example word "fantastic" appears 80 times in positives and 20 times in negatives so it has probability of 0.8 for word "fantastic" that sentiment is positive , after assign weight to words , you can create a bag of word of a new sentiment, ignore non positive and negative words , calculate probability of that sentence for each positive and negative word as follows :
$$ p(negative|s) = p(negative)p(w_1,w_2,w_3,...,w_n) $$ $$ p(positive|s) = p(positive)p(w_1,w_2,w_3,...,w_n) $$
Which one is greater has more probability for $$ s=w_1,w_2,w_3,...,w_n $$
There are many good improvement that you can use here, bag of word ignores positions of words and dependency between them , you can don't ignore them , and you can use discriminative models over generative models(like Naive Bayes) or use some other classifiers like SVM or ... ,