A client's website was operating under a certain policy for membership sign ups for over a year. At the start of October 2012 the client implemented a new policy for sign ups that was supposed to encourage more registrations. This policy was applied site wide so there is no control group to compare against (except for the before period).
I have access to the entire history of sign ups and I am wondering if there is a test that I could use to determine if there is a statistically significant difference in the number of registrations before and since the policy change? Could I do something as simple as average sign ups per day before and after the policy change and do a t-test?
Best Answer
You are describing "intervention analysis" or "interrupted time series". It refers to estimating how much an intervention has changed a time series. (Intervention-analysis is even one of the tags here, so I am proposing an edit to add it to your question.)
Among other ways, it can be done using an autoregressive integrated moving average (ARIMA) model. ARIMA should be done on a stationary time series but you can estimate a seasonal component and control for it if necessary. And NickAdams is right that you don't want to use raw numbers but rather use proportion of visitors who sign up.