This is my first question on stats, just trying to learn the basics of time series analysis with R. So any good suggestions about learning resources will be highly appreciated as well as the answer to the question.
For the data below, let's say it represents number of website visits per day, I would like to find out:
-
What the weekly pattern is (e.g. the highest number of visits
occurs on Thursdays, the lowest on Fridays etc.) -
Automatically detect changes in that pattern (e.g. in 2008 most of the visits
occur on Thursdays, but the in 2009-01-04 the pattern changes to
something else)
Please let me know if I can provide more details.
> str(daily)
An ‘xts’ object on 2007-02-19 23:32:16/2013-05-05 15:09:17 containing:
Data: num [1:2268, 1] 55 32 70 48 75 50 48 46 36 55 ...
- attr(*, "dimnames")=List of 2
..$ : NULL
..$ : chr "cnt"
Indexed by objects of class: [POSIXct,POSIXt] TZ:
xts Attributes:
NULL
> head(daily)
cnt
2007-02-19 23:32:16 55
2007-02-20 23:58:58 32
2007-02-21 23:40:41 70
2007-02-22 23:01:41 48
2007-02-23 23:53:06 75
2007-02-24 23:47:07 50
plot(daily)
Full dataset: https://dl.dropboxusercontent.com/u/65347419/daily.csv
Best Answer
You will need to consider 6 daily dummies, 11 monthly dummies, your ~10-15 holiday dummy variables. You will need to NOT consider any ARIMA as you want to rely more upon deterministic variables already listed. You will need to also consider trend(a dummy variable like 1,2,3,4,5,6etc, and perhaps changes in trend so there could be multiple so 0,0,0,0,1,2,3,4,5,,etc.), outliers, levels shifts, changes in seasonality (ie seasonal pulses as you very smartly point out that there are changes in the day of the week pattern!!!), lead and lag impacts around the holidays. There might also be day of the month variables, but we see that more with datasets tied to cash as payday is usually around the end of the month and middle of the month.
You would need to remove which dummy variables are insignificant. You can do a poor man's check when you are done to compare the coefficient in the model vs a % of the total to see if they make sense. For example if Monday contributes 50% of the overall volume then your Monday dummy should be POSITIVE and much larger than the other cooefficients.
Feel free to post your data and I would be glad to look at it. Just make sure to state the beginning observations date and the country where the data is from in order to bring in the appropriate holidays.
We have been working on time series(since 1975) and the issue of daily data(since 1998).
You will need to consider 6 daily dummies, 11 monthly dummies, your ~10-15 holiday dummy variables. You will need to NOT consider any ARIMA as you want to rely more upon deministic variables already listed. You will need to also consider trend(a dummy variable like 1,2,3,4,5,6etc, and perhaps changes in trend so there could be multiple so 0,0,0,0,1,2,3,4,5,,etc.), outliers, levels shifts, changes in seasonality (ie seasonal pulses as you very smartly point out that there are changes in the day of the week patn!!!), lead and lag impacts around the holidays. There might also be day of the month variables, but we see that more with datasets tied to cash as payday is usually around the end of the month and middle of the month.
You would need to remove which dummy variables are insignificant. You can do a poor man's check when you are done to compare the coefficient in the model vs a % of the total to see if they make sense. For example if Monday contributes 50% of the overall volume then your Monday dummy should be POSITIVE and much larger than the other cooefficients.
Feel free to post your data and I would be glad to look at it. Just make sure to state the beginning observations date and the country where the data is from in order to bring in the appropriate holidays.
We have been working on time series(since 1975) and the issue of daily data(since 1998).
Yes, I appreciate your goal of learning how to get do this, but the best I can do is this. Maybe you can reverse engineer?
Sorry for the delay!
Ok, we have analyzed your data and here are our findings. We reduced the data set to use the last 1,162 observations. The data begins on Monday 3/1/2010. The Monday start date is very important when inpreting the day of the week variables.While 6 years of data can be be helpful, in this case it is too much data as the data is so small at the beginning.
Here is a summary of the average and holidays:
The average demand is 211.
Let's review the holidays, there is a decrease in demand starting 4 days before Christmas of 48.29. Thanksgiving has an impact on the day of and the day af. Most holidays have a negative impact except St.Patrick's.
Y(T) = 211.59
+[X1(T)][(- 48.2988B*-4- 59.9340B*-3- 100.12 B*-2
- 238.07 B*-1- 150.08 - 352.52 B** 1)] M_CHRISTMAS +[X2(T)][(- 64.4805)] M_CINCODEMAYO +[X3(T)][(- 15.8391)] M_COLUMBUS +[X4(T)][(- 31.9900B*-3- 100.62 - 11.5661B* 1
- 27.8408B** 2- 14.4484B** 4)] M_GOODFRIDAY +[X5(T)][(- 14.9771B** 1)] M_FATHERSDAY +[X6(T)][(- 36.0068B** 1)] M_HALLOWEEN +[X7(T)][(- 195.05 - 103.42 B** 1)] M_JULY4TH +[X8(T)][(- 198.80 - 53.2202B** 1)] M_LABORDAY +[X9(T)][(- 28.3956- 29.5278B** 1)] M_MARDIGRAS +[X10(T)[(- 81.8183)] M_MARTINLKING +[X11(T)[(- 209.58 - 19.8445B** 1)] M_MEMORIALDAY +[X12(T)[(- 166.62 B*-4- 82.5935B*-3- 73.4411B*-2
- 218.53 B*-1- 117.53 - 115.39 B** 1)] M_NEWYEARS +[X13(T)[(- 113.83 - 12.5742B** 1)] M_PRESIDENTS +[X14(T)[(+ 13.2287B** 1)] M_STPATRICKS +[X15(T)[(- 37.4732B** 1)] M_STVALENTINES +[X16(T)[(- 244.69 - 206.13 B** 1)] M_THANKSGIVI +[X17(T)[(- 42.7715+ 17.9379B** 4+ 13.8972B** 5)] M_VEANSDAY
Autobox searches for impacts when Holidays land on a Monday or a Friday. The Monday_after a holiday on a Friday is negative 57.23. When there is a holiday on a Friday or Monday the weekend had a lower demand of 3. +[X18(T)[(- 57.2342)] MONDAY_AFTER +[X19(T)[(- 3.4639)] LONGWEEKEND
The month of the year pattern has February and March as the largest months and August as the lowest month. February is not significant so it is the same as the average. March is the intercept.
Saturdays are the lowest and Sundays(not shown as it is the intercept or average of 211.59) are at the average and Tuesdays and Wednesdays Remember that Monday was the first day of in the dataset so the first variable reflects Monday.
Two time trends. The first begins at time period 1 and indicates an increase of volume each day by .752. The second trend is negative at -.630 and starts at period 583, but the in general the trend is still up (ie .752-.630=+.122).
There are 22 one-time (pulse) outliers and 3 level shifts(changes in the intercept) 9 seasonal pulses reflecting a change in the day of the week pattern.
It looks like day 6 and 7(sat and sun) have evolved to be lower a couple of times. There was a drop found on Saturdays beginning 1/15/2011, 1/29/2011, 3/5/2011, and 2/11/2012. Sundays also had some similar drops. Day 2(Tuesdays) also had an increase beginning 5/15/2012 of +27.8687.
Four level shifts occurred with a decrease of 4.44 beginning 10/25/2010, a decrease of 32.45 beginning 8/3/2011, a decrease of 64.65 beginning 4/13/2011 and an increase of 35 beginning 3/19/2012.
If you do simple math and look at the contribution of a month to total or the day to the total you will see that the coefficients are similar. See the XLS file for the check on this. It shouldn't be exact, but rather directional in nature and it is.
We did allow Autobox to search for arima, special days of the month as this is not CASH demand related to pay days, and expanded the number of outliers to be searched or to a max of 100 due to the largest sample size.
You can see the output from the Autobox run and the XLS file showing the "poor man's" model to compare to the coefficients from Autobox in Dropbox here https://www.dropbox.com/sh/fyd0lvbnjrlbwoz/M0sH1FFhTu