Solved – How run Random Forest when there is temporal structure in the data

random foresttime series

I am used to data sets that dont have a time component. In
In reading up on time series data i learned the importance of transforming the data into stationary data before applying the ARIMA model to univariate time series. But what if i have a multivariate data set (meaning: many features potentially explaining one response variable) and i would like to use an advanced ML technique such as a regression random forest? Before applying the random forest, should i first transform the data into a stationary time series?

Thanks!

More details with specific questions below:

-for each point in time ( t=0, 1,….T) i have a value for the response variable y and the features x1, x2….xN)
-I need to predict future values of y based on knowing x1, x2,…xN
-note:i cannot use time as one of the features in predicting y
-by the way: if i disregard any temporal structure in the data, and run the random forest on a randomly chosen training data set, and evaluate the error on the remaining test set i get a pretty good result. Note: since i disregarded the temporal order, my test set observations do not necessarily occur after my training set observations.
Questions:
1-Would i maybe get better results if i turned the data into stationary data and then ran the random forest on the transformed data?
2-If so, do i need to apply the same transformation to the y as well as to the x variables? E.g. If to remove a temporal structure in the y i difference the y and remove a seasonal component in the y, do i need to also difference all the x and remove any seasonal component in the x before running random forest?
3-once i have made the series stationary, is it ok to randomly pick a training and test data set without respecting temporal order (ie without having test set observations occur after training set observations)?
e features.

Best Answer

To apply Random Forest you dont need to check for any assumption. Take y=t,and x=t-1, t-2, t-3 (all lags you feel would help).
But instead of applying RF etc, go with Time series techniques like- Hybrid Model in R, which will give you ensemble of ARIMA, ETS, NN, TBATS, THETAM, STLM algorithms.

Another algo that handles multiple level of seasonality ( Facebook's Prophet model)-

https://machinelearningstories.blogspot.in/2017/05/facebooks-phophet-model-for-forecasting.html

Related Solutions

Solved – Random Forest Regression Overfitting – Quantile Test on Test Data

"nodesize: Minimum size of terminal nodes. Setting this number larger causes smaller trees to be grown (and thus take less time). Note that the default values are different for classification (1) and regression (5)."

Check out this parameter in the randomForest package. By upping the number of observations in terminal nodes you will have smaller (in terms of depth) trees. "Over-fit" trees are generally big (deep in terms of depth).

Solved – Using a Random Forest for Time Series Data

It works well but only if the features are properly prepared so that the order of the lines is not important anymore.

E.g. for a univariate time series $y_i$, you would use $y_i$ as response and e.g. the following features:

Lagged versions $y_{i-1}$, $y_{i-2}$, $y_{i-3}$ etc.
Differences of appropriate order, e.g. $y_{i-1} - y_{i-2}$, $y_{i-1} - y_{i-8}$ (if there is weekly seasonality expected and the observations occur daily) etc.
Integer or dummy coded periodic time info such as month in year, week day, hour of day, minute in hour etc.

The same approach works for different modelling techniques, including linear regression, neural nets, boosted trees etc.

An example is the following (using a binary target "temperature increase" (y/n)):

library(tidyverse)
library(lubridate)
library(ranger)
library(MetricsWeighted) # AUC

# Import
raw <- read.csv("https://raw.githubusercontent.com/jbrownlee/Datasets/master/daily-min-temperatures.csv")

# Explore
str(raw)
head(raw)
summary(raw)
hist(raw$Temp, breaks = "FD")

# Prepare and add binary response
prep <- raw %>% 
  mutate(Date = ymd(Date),
         y = year(Date),
         m = month(Date),
         d = day(Date),
         increase = 0 + (Temp > lag(Temp)))

with(prep, table(y))
summary(prep)

# Plot full data -> year as seasonality
ggplot(data = prep, aes(x = Date, y = Temp))+
  geom_line(color = "#00AFBB", size = 2) +
  scale_x_date()

# No visible within year seasonality
prep %>% 
  filter(y == 1987) %>% 
ggplot(aes(x = Date, y = Temp))+
  geom_line(color = "#00AFBB", size = 2) +
  scale_x_date()

# Add some lags and diffs & remove incomplete rows
prep <- prep %>% 
  mutate(lag1 = lag(Temp),
         lag2 = lag(Temp, 2L),
         lag3 = lag(Temp, 3L),
         dif1 = lag1 - lag2,
         dif2 = lag2 - lag3) %>% 
  filter(complete.cases(.))

# Train/valid split in blocks
valid <- prep %>% 
  filter(y == 1990)
train <- prep %>% 
  filter(y < 1990)

# Models
y <- "increase" # response
x <- c("lag1", "lag2", "lag3", "dif1", "dif2", "y", "m", "d") # covariables
form <- reformulate(x, y)

# Logistic model: Linear dependence between difs and lags
fit_glm <- glm(form, 
               data = train, 
               family = binomial()) 
summary(fit_glm)

# Random forest
fit_rf <- ranger(form, 
                 data = train,
                 seed = 345345, 
                 importance = "impurity", 
                 probability = TRUE)
fit_rf
barplot(-sort(-importance(fit_rf))) # Variable importance

# Evaluate on 1990 for glm by looking at ROC AUC
pred_glm <- predict(fit_glm, valid, type = "response")
AUC(valid[[y]], pred_glm) # 0.684 ROC AUC

# Then for rf
pred_rf <- predict(fit_rf, valid)$predictions[, 2]
AUC(valid[[y]], pred_rf)    # 0.702 ROC AUC

# view OOB residuals of rf within one month to see if structure is left over
random_month <- train %>% 
  mutate(residuals = increase - fit_rf$predictions[, 2]) %>% 
  filter(y == 1987, m == 3) 

ggplot(random_month, aes(x = Date, y = residuals))+
  geom_line(color = "#00AFBB", size = 2) +
  scale_x_date()

Replacing variables "y" and "m" by factors would probably improve the logistic regression. But since the question was about random forests, I leave this to the reader.

Best Answer

Related Solutions

Solved – Random Forest Regression Overfitting – Quantile Test on Test Data

Solved – Using a Random Forest for Time Series Data

Related Question