Solved – Neural network language model – prediction for the word at the center or the right of context words

language-modelsnatural language

Neural network language model – prediction for the word at the center or the right of context words?

On Bengio's paper, the model predicts probability by n words for the next word, like predicting probabilities of "book", "car", etc., by n words before it, like "this", "is", "a", "good". However, in tagging problems in NLP, like those in Collobert's papers, a common setup, the window model, is tag prediction for the center word by surrounding words.

Are there some studies on neural network language models for prediction of word probabilities at the center by surrounding words, like predicting probabilities of word at the center like "a", "the" by context words "this", "is" (at the left) and "good", "car" (at the right)?

Best Answer

The task of finding missing words in a text sometimes referred to as text imputation, or sentence completion.

One paper exploring it with ANN: Solving Text Imputation Using Recurrent Neural Networks. Arathi Mani. CS224D report. 2016. http://cs224d.stanford.edu/reports/ManiArathi.pdf

In this paper, we have shown that the bidirectional RNN yields the best Levenshetein and perplexity scores out of the three models tested for our missing data problem where we try to impute a single word into a sentence that is missing exactly one word from an unknown location.

One paper comparing several approaches including RNN: Zweig, Geoffrey, John C. Platt, Christopher Meek, Christopher JC Burges, Ainur Yessenalina, and Qiang Liu. "Computational approaches to sentence completion." In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pp. 601-610. Association for Computational Linguistics, 2012. https://scholar.google.com/scholar?cluster=4615153328130310080&hl=en&as_sdt=0,22 ; http://www.aclweb.org/anthology/P/P12/P12-1063.pdf

This paper studies the problem of sentencelevel semantic coherence by answering SAT-style sentence completion questions

Related Question