Solved – Understanding LDA inference

Question

It is said that the key inferential problem that needs to be solved to use LDA (latent Dirichlet allocation) is that of computing the posterior distribution $p(\theta,z | w, \alpha ,\beta)$. I know LDA inference was first presented using variational inference on a simplification of LDA's graphical model, but other methods such as Gibbs Sampling allow to estimate $p(\theta,z | w, \alpha ,\beta)$.

After calculating $p(\theta,z | w, \alpha ,\beta)$, how is it used afterwards? How can we do document classification with $p(\theta,z | w, \alpha ,\beta)$?

(Notation: the same in the original LDA paper and on Wikipedia.)

Answer 1

How to classify documents

Importantly, latent Dirichlet allocation is an unsupervised method: On its own, it doesn't account for the class or category of a document. But, as discussed in section 7.2 of the paper that introduced it, it can be used to develop features for classification:

A challenging aspect of the document classification problem is the choice of features. Treating individual words as features yields a rich but very large feature set (Joachims, 1999). One way to reduce this feature set is to use an LDA model for dimensionality reduction. In particular, LDA reduces any document to a fixed set of real-valued features—the posterior Dirichlet parameters $\gamma\ast(\textbf{w})$ associated with the document.

So as a general, practical answer to your second question: Parameters of the topic distribution for a document can be used as features in a classifier of your choice. That's exactly what the authors of LDA did in their experiments:

In these experiments, we estimated the parameters of an LDA model on all the documents, without reference to their true class label. We then trained a support vector machine (SVM) on the low-dimensional representations provided by LDA and compared this SVM to an SVM trained on all the word features.

Here's an example of what this could look like in python. It transforms the digits dataset from sklearn to a 16-topic space, then predicts the digit using logistic regression. (Sixteen chosen rather arbitrarily after some exploration in my answer here.)

# -*- coding: utf-8 -*-
"""
Created on Fri May 27 15:24:16 2016

@author: SeanEaster
"""

from sklearn.decomposition import LatentDirichletAllocation as LDA
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.cross_validation import train_test_split
from sklearn.metrics import confusion_matrix

import numpy as np

digits = load_digits()

images = digits['images']
images = [image.reshape((1,-1)) for image in images]
images = np.concatenate(tuple(images), axis = 0)

lda = LDA(n_topics = 16) 

X = lda.fit_transform(images)
Y = digits['target']

xTrain, xTest, yTrain, yTest = train_test_split(X,Y,test_size =.2, random_state=9)

classifier = LogisticRegression(C = 1e5) # Choice of C here is arbitrary; in practice, cross validate
classifier.fit(X,Y)
print confusion_matrix(yTest, classifier.predict(xTest))

Which gives reasonable results—here's the confusion matrix it prints:

[[33  0  0  0  0  0  0  0  0  0]
 [ 0 36  1  0  0  0  0  0  2  1]
 [ 0  1 40  2  0  0  0  0  2  0]
 [ 0  0  0 32  0  0  0  0  0  2]
 [ 0  0  0  0 36  0  0  4  0  1]
 [ 0  0  0  1  0 34  0  0  0  4]
 [ 0  0  0  0  0  0 29  0  0  0]
 [ 0  0  0  0  0  0  0 27  1  0]
 [ 0  6  1  0  1  1  0  1 25  1]
 [ 0  0  0  1  1  1  0  0  3 29]]

For a text application, see this classification example from the sklearn docs.

Uses for the posterior distributions

To your first question, there are still uses for LDA topics outside of classification, namely that extracted topics can give a descriptive summary of a corpus. Another sklearn example does this on the 20 newsgroups dataset, and prints the top words of topics. Here's it's output:

Topics in LDA model:
Topic #0:
government people mr law gun state president states public use right rights national new control american security encryption health united
Topic #1:
drive card disk bit scsi use mac memory thanks pc does video hard speed apple problem used data monitor software
Topic #2:
said people armenian armenians turkish did saw went came women killed children turkey told dead didn left started greek war
Topic #3:
year good just time game car team years like think don got new play games ago did season better ll
Topic #4:
10 00 15 25 12 11 20 14 17 16 db 13 18 24 30 19 27 50 21 40
Topic #5:
windows window program version file dos use files available display server using application set edu motif package code ms software
Topic #6:
edu file space com information mail data send available program ftp email entry info list output nasa address anonymous internet
Topic #7:
ax max b8f g9v a86 pl 145 1d9 0t 34u 1t 3t giz bhj wm 2di 75u 2tm bxn 7ey
Topic #8:
god people jesus believe does say think israel christian true life jews did bible don just know world way church
Topic #9:
don know like just think ve want does use good people key time way make problem really work say need

You can already see some intuitive overlap with the newsgroup names, described here, e.g. talk.politics.guns, talk.religion.misc. You can carry this descriptive analysis further, but exactly how depends much on your interest.