Up to a certain point: memory. If you do not have enough memory, then you will either fail or you will swap to disk, which is slow. However, once you have enough memory to hold your data, then additional memory only helps to the extent that your operating system uses the extra to cache files (or to run anti-virus, or to run your web browser...)
If you are reading video from files, then file access is often the limiting factor. If you are using multiple streams of live video then USB bandwidth and contention is often the limiting factor.
Once you have an image, then it depends what you are going to do with the image. There are some operations for which automatic parallelization is used, but other common operations on video turn out to be relatively serial. Using parallel computing toolbox to process multiple files automatically might or might not help, depending what is being done. Mostly using parfor on different files helps only to the extent of at most two workers per drive and sometimes only at most two workers per drive controller (beyond that you just enter into drive contention.)
For operations that are not automatically parallizable, then typically single core speed is most important than number of cores. Beyond around 6 or so cores it is not uncommon for communications overhead to start to overwhelm the benefit of adding additional cores.
If you are thinking of doing Deep Learning, then which Deep Learning can be important. Some of the Deep Learning routines benefit strongly from GPU.
Best Answer