fminsearch can end up seaching indefinitely if any nan or inf values are encountered.
To prevent this, you need to use an options structure that has an OutputFcn. The OutputFcn should accept 3 inputs, the first of which is the current x values, the second of which is a struct with information about the optimization, and the third is some flag information I think. You can test ~isfinite(optimValues.fval) to see if you have encountered inf or nan, and return true if you want to stop in that case.
Note that fminsearch provides no way to reject a particular value and keep going: it only provides a way to stop. The function value it will return will be that nan or inf. There are tricks you can use to be more useful, but it is not as convenient as one might hope.
Best Answer