So I have this data set of 56 users with 52 weeks worth of weekly average data for blood pressure and exercise level recordings. I would like to use change point analysis (https://sites.google.com/site/changepointanalysis/) in R to understand where changes are occurring. However to make CPA usable, the observations need to be independent (at least have no strong autocorrelation).
I performed ACF and the Ljung-box test for lags up to 20, and there is autocorrletion in many of the time series.
So my question is, what is an appropriate way to remove this?
I've looked around and have found some possibilities, but nothing has made it overtly clear why I would choose one over another, or what is a lowest risk approach.
Various possibilities I have seen (some from this site):
- Low pass filter, inverse subtraction
- First differences
- Detrending
- Seasonal adjustment
- Data transformation (e.g. convert difference operator into ratio)
- Exploratory data analysis (EDA) smoothing techniques
One of the people at work said something about using a low pass filter, using inverse subtraction to remove correlation, and then finding the frequency through fast fourier transform or spectrogram analysis. I'm not sure about all that.
I really appreciate any comments, I'm kind of lost at a crossroads right now. Thanks!
Best Answer
First there are ways to make a non-stationary series stationary. 1. Remove polynomial trends (e.g. 1st, 2nd,3rd order differencing). 2. Remove seasonal components (e.g. seasonal differencing) 3. Identify time changes (intervention analysis)
These are not the only approaches but are what you will find suggested by Box, Jenkins and Reinsel in their time series book.
Those steps will make the series stationary but not necessarily independent or uncorrelated.
After doing the above an autoregressive moving average time series model may then leave the residuals uncorrelated (of course uncorrelated does not imply that the residuals are independent).
This is often called the Box-Jenkins approach to time series analysis. Usually the interest is in the model that characterizes the data and not the uncorrelated residuals.
This is all done in the time domain. Periodic components can also be identified in the frequency domain by taking a Fourier transformation and then smoothing the periodogram after stationarity is observed.
Rather than removing trends and autocorrelation intervention analysis will identify change points as part of the modeling process. Software such as autobox will do this automatically for you. Irishstat can help you with that.
Keep in mind that lack of correlation is not the same as independent. Also stationarity means that the series does not change behavior in time. This means no trends, seasonal components or changes in variance with time. Also stationary series can have significant autocorrelation and filters or ARMA models need to be applied to stationary series to get uncorrelated residulas.