Solved – Are there any versions of t-SNE for streaming data

Question

My understanding of t-SNE and the Barnes-Hut approximation is that all data points are required so that all force interactions can be calculated at the same time and each point can be adjusted in the 2d (or lower dimensional) map.

Are there any versions of t-sne that can efficiently deal with streaming data? So if my observations are arriving one at a time, it will find the best location on the 2d map to place the new observation, or continuously update all points on the 2d map to account for ht new observation.

Would this even make sense or does it go against the setup of t-sne.

Answer 1

I had exactly the same question and posted it on a YouTube video of a CS231n lecture given by Andrej Karpathy a few weeks ago. Here is the question I posted followed by Andrej' response:

https://www.youtube.com/watch?v=ta5fdaqDT3M&lc=z12ji3arguzwgxdm422gxnf54xaluzhcx

Q:

Does t-SNE need an entire batch of images (or more generally, data) to create the low-dimensional feature space? With PCA you can create a low-dimensional feature space on a batch of data and then project new data points onto that same space without having to "retrain". Is that true for t-SNE?

I ask because I noticed that scikit-learn has t-SNE as part of its manifold class, but that module does not have a transform() method as PCA does. So, at least, in sklearn, it would seem this is not possible.

My question boils down to this. How would you apply t-SNE in a streaming or online situation where you want to continually update the visualization with new images? Presumably, one would not want to apply the algorithm on the entire batch for each new image.

A:

+Evan Zamir yes this is possible with t-SNE, but maybe not supported out of the box with regular t-SNE implementations. Normally each point's location is a parameter in the optimization, but you can just as well create a mapping from high-D -> low-D (e.g. neural net) and backprop through the locations. Then you end up with the embedding function and can project new points. So nothing preventing this in principle, but some implementations might not support it as it's a less frequent use case.