I have been working on an assignment where I train upon 3 corpora in 3 separate languages, and then I read in a set of sentences and use a number of models to determine the most likely language for each sentence. That part I have done. (for reference: the models I implemented were a Bigram Letter model, a Laplace smoothing model, a Good Turing smoothing model, and a Katz back-off model).
Now, I am tasked with trying to find the perplexity of the test data (the sentences for which I am predicting the language) against each language model. I have read the relevant section in "Speech and Language Processing" by Jurafsky and Martin, as well as scoured the internet to try to figure out what it means to take the perplexity in the manner above.
The examples I consistently see are not helpful at all. I have zero understanding as to what I need to be doing here. What does it mean to take the perplexity of test data? How do the language models play into it? How do the separate languages themselves factor into things?
Simply knowing an equation does me no good if I don't understand how to implement it.
Best Answer
Your language models can be used to estimate the probability of observing each token in the test data. If you average over all the tokens then you will have the entropy of the test data. The perplexity is just $2^\text{entropy}$