I am using moving window technique for data analysis…
For example I compute the mean, the standard deviation and etc. for a given window.
And I wonder if there's any good criterion to determine window size…
I thought of computing multivariate variance of mean, sd, and etc. and
select the window size that minimize the variance…
But for the most of cases, the longer the window size, the smaller the variance…
So I am stuck… Is there any criterion that can solve the problem of determine the
window size or I have to go with my eyes and pick one that best entertain my eyes?
Best Answer
I agree with whuber. You should have some methodology, for example bootstrapping and stability criterion (that is perturb your data a little bit and check that your estimates do not change too much), or better sharp theoretical results, to help you decide if you have enough samples to compute your estimates correctly.
Then, you should use the minimum amount of data that provides you strong guarantee (theoretical or empirical) on the relevance of your results, but no more as you will smooth the signal (or violate even more the stationary hypothesis).
As long as you stay with estimating mean, variance, and so on, you should be able to find theoretical results and guidelines. If you want to determine the minimum length of the window for a complex processing (say machine learning algorithms), you should go for an empirical study, cf. this study for an example on the clustering of correlated random variables.