forecastinggoodness of fittime series
I forecasted/simulated a time series with a n step ahead forecast (n-ahead = 250) with 4 different time series models. Now I want to test, which of these models fits the best for the data.
All I found were methods, were we need the actual future values to compare (e.g RMSE, $R^2$ etc.). Are there techniques to test for a goodness of fit without these future values? Also I am interested in methods to compare the different forecasts.
Your data is discrete and as such ARIMA modelling can be useful but tentative at best. See this post What are the consequences of not meeting the assumptions for the residuals of ARIMA model? which discusses discrete data analysis with a slightly different twist.
I took your daily data (365 values) and used AUTOBOX to detect structure. There is no identifiable autoregressive memory structure (ARIMA) . There is no significant day of the week effect in the observations but curiously outliers are not equally distributed across days suggesting a possible special cause. There is however two months of the year that appear to be statistically significant although with only 1 year of data that could be proxying something else. No particular day of the month appeared to be significant. There are a number of days that appear to be "unusual" i.e. outliers. You might examine those dates in order to find a possible assignable cause as part of you discovery process.
Here is the Actual data 
. The equation 
and the Actual , Fit and Forecast graph is here 
. Notice that while the expected value is flat for the next 30 days the upper confidence limits (obtained through monte-carlo methods by resampling the decidely non-normal residuals) are not. Standard Box-jenkins confidence limits for forecasts assume perfect kdnowlede of the estimated parameters and normality of the errors providing possibly naive symmetric limits with the proviso that unusal values may have occurred in the past but they won't reoccur. Lower limits are clearly 0.0 by specification and are shown here for generality purposes.
It turns out that while you have a significant # of outliers they are not evenly distributed across days. Since the confidence limits include the very possibility of future shocks (anomalies) they are "jagged" . Here is the forecast plot by itself .
This is a very broad question. I'll comment on a few points.
What do you mean by "I put zero values for the sales in that new dates that I've just created"? This sounds like you fill in zeros at the end for series that ended before a common end date - and then you'd attempt to forecast these zero sales for products that have been delisted. This wouldn't make sense, so I'm sure I'm misunderstanding something.
You should think long and hard about whether filling in NAs with zeros or using loocf (as below) makes sense. Maybe some of your products are seasonal and are simply not sold in summer or in winter? If so, filling in makes no sense. Yes, in such a case ARIMA will have problems, although AFAIK auto.arima() uses some kind of a state space approach that can deal with NAs. You may want to look at fitting simple linear models with a trend regressor and multiple seasonal dummies.
This looks fine to me.
I assume you are using the frequency parameter to tell R that you can have a 52-period seasonality?
It's unclear to me what kind of regressors you are setting up here. Prices? Promotions?
I don't think this makes sense. In step 8 below, you fit a regression with ARIMA errors, to take seasonality, autoregression and MA behavior into account, so you already believe that an OLS model is wrongly specified. So why fit it? Better to skip steps 4 and 5 and evaluate the regressions with ARIMA models directly on the holdout sample.
See above. Information criteria have only a tenuous relationship to forecasting accuracy, if that's what you are after.
Think about where your zeros come from. (See above.) Are your retail sales per store, or aggregated across multiple stores? If the latter, you shouldn't have too many zeros... except for delistings, again see above. If you have really low volume data, because of extreme slow movers, then regression/ARIMA doesn't make a lot of sense, because it inherently presupposes continuous data. I recently wrote a little article on forecasting count data in retail (Kolassa, 2016, International Journal of Forecasting, which may be interesting to you.
Taking logs may or may not be worthwhile.
I'd encourage you to look at each outlier separately and think about what may have caused that outlier. Data errors? Someone buying a heap of product? (That should only show up in a single store and even out if you are looking at aggregate data.) Plus, are you considering your regressors in assessing outliers? An "outlier" may simply be the effect of a successful promotion. Needless to say, you don't want to remove or "correct" these data, but capture and forecast the promotion effect.
See above. This should really be the core of your process. You can iterate the fitting over different subsets of regressors and evaluate each model. If you can't fit an ARIMA model, it makes sense to look at the time series, and the residuals from the regression, and try to understand what's happening. Sometimes you do run into numerical difficulties in fitting. In such a case ets() is a good fallback solution, or you could first run a regression on your regressors, then feed the residuals into ets().
See above. Logs may or may not improve your forecasts.
This makes sense, of course. mae is usually good. Also look at the distribution of residuals, especially if you have low volume count data (see the paper I referenced for details), and at whether your forecasts are systematically biased.
You may want to browse through some previous questions on forecasting in retail.
Best Answer
In-sample fit is a very bad proxy for out-of-sample accuracy, and optimizing in-sample fit will tempt you to overfit.
The gold standard is to use a holdout sample: hold back the last observations of your historical data, fit your models to the remaining data, then forecast into the holdout sample and see which model performs best on it. This will not help you much if you need a long-range forecast, with a horizon longer than your history - but "standard" time series forecasting algorithm are not really made for such long-range forecasts, anyway.
If you compare models from the same model class, you can use information criteria such as AIC or BIC, which trade off in-sample fit against model complexity. So you can compare different ARIMA models, or different ETS models. However, you cannot compare an ARIMA model against an ETS model using information criteria.
I recommend Forecasting: Principles and Practice by Rob J Hyndman and George Athanasopoulos.