machine learning
I am looking at the AUCs of three RandomForestClassifier models. Before this, I split my data using test and train using a random_state.
When I change the random_state for the data split, the AUCs changes. Is this supposed to happen or does it mean my model most likely needs parameter tuning?
Here are the AUCs using a random_state of 1 and then a random_state of 2.
Model A. 0.76 -> 0.71
Model B. 0.73 -> 0.69
Model C. 0.57 -> 0.58
Thanks.
Because the ML algorithms works minimizing the error on the training, the expected accuracy on this data would be "naturally" better than your test results. Effectively when the training error is too low (aka accuracy too high) maybe there is something that has gone wrong (aka overfitting)
As suggested by user5957401, you can try to cross-validate the training process. For example, if you have a good amount of instances, a 10 fold cross-validation would be fine. If you need also to tune hyper parameters, a nested-cross validation would be necessary.
In this way the estimated error from the test-set will be "near" the expected one (aka, the one that you'll get on real Data). In this way, you can check if your result (AUC 0.80 on the test set) is a good estimate, or if you got this by chance
You can try also other techniques, like shuffling several times your data before the cross-validation task, to increase the result reliability.
I took your 201 monthly values and examined them in an automatic mode with AUTOBOX , a time series analysis package that I helped to develop. The model developed is here 
. The Actual,Fit and Forecast graph is here 
and the ACF of the residuals is here 
. The forecast plot for the next 12 periods is here.
.
There are a number of anomalous data points that will thwart any simple brute force attempt to examine candidate arima structure . auto.arima as it is not robust to latent anomalies (pulses,step shifts, seasonal pulses, local time trends ) and thusly over differences and over paramaterizes arima structure induced by the over differencing.
In terms of your "complaint" about the size of out-of-sample mapes , it all has to do with forecasting "small numbers" as was pointed out by @jbowman .
Finally the plot of the model residuals 
markedly more visually acceptable than yours.
In closing .... to improve the accuracy of any model "Know the Assumptions" as A. Wald wisely reflected.
Best Answer
YES
When you have one random state, you select certain data for training, develop a model, and then test the model on a holdout data set.
When you change the random state, you select different data for training, develop a different mode, and then test this other mode on a different holdout data set.
I would expect different models that are tested on different data sets to have different performance.