Solved – Term frequency/inverse document frequency (TF/IDF): weighting

data miningfeature selectionr

I've got a dataset which represents 1000 documents and all the words that appear in it. So the rows represent the documents and the columns represent the words. So for example, the value in cell $(i,j)$ stands for the times word $j$ occurs in document $i$. Now, I have to find 'weights' of the words, using tf/idf method, but I actually don't know how to do this. Can someone please help me out?

Best Answer

Wikipedia has a good article on the topic, complete with formulas. The values in your matrix are the term frequencies. You just need to find the idf: (log((total documents)/(number of docs with the term)) and multiple the 2 values.

In R, you could do so as follows:

set.seed(42)
d <- data.frame(w=sample(LETTERS, 50, replace=TRUE))
d <- model.matrix(~0+w, data=d)

tf <- d
idf <- log(nrow(d)/colSums(d))
tfidf <- d

for(word in names(idf)){
  tfidf[,word] <- tf[,word] * idf[word]
}

Here's the datasets:

> colSums(d)
wA wC wD wF wG wH wJ wK wL wM wN wO wP wQ wR wS wT wV wX wY wZ 
 3  1  3  1  1  1  1  2  4  2  2  1  1  3  2  2  2  4  5  5  4 
> head(d)
  wA wC wD wF wG wH wJ wK wL wM wN wO wP wQ wR wS wT wV wX wY wZ
1  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  1  0  0
2  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  1  0
3  0  0  0  0  0  1  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
4  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  1  0  0  0
5  0  0  0  0  0  0  0  0  0  0  0  0  0  1  0  0  0  0  0  0  0
6  0  0  0  0  0  0  0  0  0  0  1  0  0  0  0  0  0  0  0  0  0
> head(round(tfidf, 2))
  wA wC wD wF wG   wH wJ wK wL wM   wN wO wP   wQ wR wS wT   wV  wX  wY wZ
1  0  0  0  0  0 0.00  0  0  0  0 0.00  0  0 0.00  0  0  0 0.00 2.3 0.0  0
2  0  0  0  0  0 0.00  0  0  0  0 0.00  0  0 0.00  0  0  0 0.00 0.0 2.3  0
3  0  0  0  0  0 3.91  0  0  0  0 0.00  0  0 0.00  0  0  0 0.00 0.0 0.0  0
4  0  0  0  0  0 0.00  0  0  0  0 0.00  0  0 0.00  0  0  0 2.53 0.0 0.0  0
5  0  0  0  0  0 0.00  0  0  0  0 0.00  0  0 2.81  0  0  0 0.00 0.0 0.0  0
6  0  0  0  0  0 0.00  0  0  0  0 3.22  0  0 0.00  0  0  0 0.00 0.0 0.0  0

You can also look at the idf of each term:

> log(nrow(d)/colSums(d))
      wA       wC       wD       wF       wG       wH       wJ       wK       wL       wM       wN       wO       wP       wQ       wR       wS       wT       wV       wX       wY       wZ 
2.813411 3.912023 2.813411 3.912023 3.912023 3.912023 3.912023 3.218876 2.525729 3.218876 3.218876 3.912023 3.912023 2.813411 3.218876 3.218876 3.218876 2.525729 2.302585 2.302585 2.525729
Related Question