Time-Series – Combining RMSE and Classification Accuracy in Loss Function

classificationfinanceloss-functionsregressiontime series

I am working on a time series regression problem applied to finance. I am interested in predicting the price change of a stock and how much will it change by.

I have framed this as a regression problem using MSE as my loss function. This gives good MSE accuracy. I would like however improve on the directional accuracy (will the price move up or down) by giving this information in the loss function (since the sign is ignored in the MSE loss function). I understand one way to improve of this is to switch to a classification problem. Switching to classification on the other hand would not give me any information on the amount of change.

Is there any way to customise the loss function in such a way so as to account for bother MSE and direction? Or is there any loss function recommended for this type of problem?

Thanks

Best Answer

One possibility would be to run two models on your time series:

  • Model 1 gives a numerical prediction of the target variable, e.g., the price change. You would evaluate this using MSE.
  • Model 2 is a classification algorithm and outputs the predicted probability of a positive price change. You would evaluate this predicted probability using the Brier score or the log loss. (The Brier score is of course nothing else than the MSE applied to a 0-1 realization, 0 if the price change is negative and 1 if it is positive.) You don't want to evaluate Model 2 predictions using accuracy, especially not in an inflationary environment - there will likely be more positive than negative price changes, and accuracy will bias your "positivity" predictions upwards.

If you then want a single KPI, you can combine the two error metrics. You would need to decide which aspect is more important to you and how much, and you would need to scale everything, since the probabilistic predictions are naturally constrained and scaled between 0 and 1, while the numerical predictions are unconstrained and scaled by the underlying asset.

The alternative I would go for would be to use probabilistic predictions for the price change, i.e., density forecasts. You can evaluate density forecasts using proper scoring rules, and given a density forecast, you can easily derive an expectation point forecast as well as a prediction of the probability the outcome is positive.

In any case, in finance you are of course always going up against the Efficient Market Hypothesis. There are many highly competent people trying to do the exact thing you are looking at, with enormous resources - and to the degree they are successful, they are destroying the signal itself. Thus, be aware that if there is a solution to your problem, somebody will likely already have found it, applied it, and thus obliterated its usefulness.

Related Question