I have a case where I want to predict a time value in minutes.
This is the problem of regression.
I also want to predict the upper bound and lower bound.
I can do it two ways:
-
Train 3 models: one for the main prediction, one for say a higher prediction and one for a lower prediction.
-
Use Quantile regression whcih gives a lower and upper bound.
However, I am not understanding how Quantile regression works.
Here is the code:
import numpy as np
import matplotlib.pyplot as plt
from sklearn.ensemble import GradientBoostingRegressor
np.random.seed(1)
#----------------------------------------------------------------------
# First the noiseless case
X = np.atleast_2d(np.random.uniform(0, 10.0, size=100)).T
X = X.astype(np.float32)
# Observations
y = f(X).ravel()
dy = 1.5 + 1.0 * np.random.random(y.shape)
noise = np.random.normal(0, dy)
y += noise
y = y.astype(np.float32)
# Mesh the input space for evaluations of the real function, the prediction and
# its MSE
xx = np.atleast_2d(np.linspace(0, 10, 1000)).T
xx = xx.astype(np.float32)
alpha = 0.95
clf = GradientBoostingRegressor(loss='quantile', alpha=alpha,
n_estimators=250, max_depth=3,
learning_rate=.1, min_samples_leaf=9,
min_samples_split=9)
clf.fit(X, y)
# Make the prediction on the meshed x-axis
y_upper = clf.predict(xx)
clf.set_params(alpha=1.0 - alpha)
clf.fit(X, y)
# Make the prediction on the meshed x-axis
y_lower = clf.predict(xx)
clf.set_params(loss='ls')
clf.fit(X, y)
# Make the prediction on the meshed x-axis
y_pred = clf.predict(xx)
# Plot the function, the prediction and the 90% confidence interval based on
# the MSE
fig = plt.figure()
plt.plot(X, y, 'b.', markersize=10, label=u'Observations')
plt.plot(xx, y_pred, 'r-', label=u'Prediction') # pred
plt.plot(xx, y_upper, 'k-') #
plt.plot(xx, y_lower, 'k-') #
plt.fill(np.concatenate([xx, xx[::-1]]),
np.concatenate([y_upper, y_lower[::-1]]),
alpha=.5, fc='b', ec='None', label='90% prediction interval')
plt.xlabel('$x$')
plt.ylabel('$f(x)$')
plt.ylim(-10, 20)
plt.legend(loc='upper left')
plt.show()
My questions are:
- How does quantile regression work here i.e. how is the model trained?
- How to use a quantile regression mode at prediction time, does it give 3 predictions, what is y_lower and y_upper?
Best Answer
To answer your questions:
When creating the classifier, you've passed
loss='quantile'
along withalpha=0.95
. You are optimizing quantile loss for 95th percentile in this situation. You can read up more on how quantile loss works here and here.In your code, you have created one classifier. You're first fitting and predicting for
alpha=0.95
, then usingclf.set_params()
you're using the same classifier to fit and predict foralpha=0.05
.For real predictions, you'll fit 3 (or more) classifiers set at all the different quantiles required to get 3 (or more) predictions.