In my test procedure I sequentially take 10 measurements of a recently perturbed physical system, and I often find the first few (between 0 and 4) measurements can be inaccurate because the system has not settled/equilibrated. The initial non-equilibrated measurements may just have less accuracy or show a distinct trend, while the remainder should not have a trend and just show random variation.
I am currently eyeballing the data to discard any initial poor data but want to have a more rigourous and automated method for selecting the "good" data. I am aware (eg here) of various outlier tests (Chauvenet, Grubbs, Pierce, Generalized ESD) but because of "masking" and trending don't think they will give reliable results. The experimental conditions are fairly well controlled so there are unlikely to be other genuinely sporadic outliers appearing.
Is there a more appropriate statistical test (than say Generalised ESD) for my procedure?
Update:
To give some more details I have run about 15 procedures so far and while some datasets have all good values most need 2 or 3 measurements dropped, so it seems reasonable to query the first 4.
The problem description above is somewhat simplified in that more than one number is determined for each measurement, ie there are several system parameters measured. Only 1 or 2 parameters show non-equilibrium effects though so there is limited scope for cross-checking.
The aim is to get reliable values for parameters of an unchanging system and therefore the model I am trying to fit is a series of constant values with some measurement error. The measuring is time-consuming but automated and can currently run overnight, the analysis is performed afterwards. It is possible to increase the number of measurements but this obviously takes more time.
From the previous results I have a good idea of the expected variation in measurements at equilibrium, so I could use that information in addition to the within-dataset variation to assess outliers. Currently I only use that information to determine if the final error is acceptable.
Six good data-points is sufficient to get an adequate estimate of the parameters and to detect any obvious problems with the process, so just dropping the first 4 is an acceptable solution. It seems from the discussion this is the best solution, unless the Dixon's test variant can be used. Where can I read more about that?
Best Answer
I don't necessarily see this as being easily treated as a time series problem. To comment on the way you detect outliers, Pierce and Chauvenet are flawed procedures that should be discussed only for historical purposes and never used. Outlier detection involves more than just knowning what the variance should be, the underlying population distribution needs to be assumed. Dixon's test and Grubbs' test assume normality and are desined for single outliers. In their original form they can be very sensitive to masking. But Dixon has variants that enable you to detect multiple outliers as long as the number of outliers is small. Also as I have mentioned in other post Dixon's test is robust to departures from normality. In your case 10 is small enough but I worry about trying to detect as many as four out of a sample of only 10. There is a little bit of a time dependence with you knowing why the outliers are likely to be among the first few measurements. But as Bill Huber pointed out in comments the sequence of 10 is too short to do any sophisticated time series modelling.
Normally I argue that outliers should not be rejected but studied further. Here you seem to have a physical reason for higher variability and or trends with the early measurements. CUSUM charts are good for detecting trends but the sequence may be too short to do much. It may be that something informal such as dropping the first four out of ten will work as a practical matter even though it is not a formal statistical test.