Now I am writing my thesis The topic is about error analysis on turbocharger test bench. In order to estimate uncertainties at test bench first of all I used Monte Carlo Method. Now I have to validate my results. I do not want to use Gauss Method. Do you have any idea how can I validate my results? Is adaptive Monte carlo method such a method to validate ? Thanks a lot.
[Math] How Can ı validate the Monte Carlo Simulation
monte carlo
Related Solutions
Yes, a simple $100(1-\alpha)\%$ confidence interval can be furnished for large sample sizes using a normal approximation; for instance, $$\hat p \pm z_{1-\alpha/2}^* \sqrt{\frac{\hat p (1-\hat p)}{n}},$$ where $\hat p$ is your $p_{\rm observed}$, the point estimate of the parameter $p$. This is not a good approximation when $p$ is close to $0$ or $1$, or when $n$ is small. In such a case, we can use alternative intervals, e.g., the Wilson score interval, or the exact binomial proportion interval using the $F$-ratio distribution. There is extensive literature on this topic.
(Incidentally, $z_{1-\alpha/2}^*$ is the $1-\alpha/2$ quantile of the standard normal distribution; i.e., it satisfies $\Pr[Z \le z_{1-\alpha/2}^*] = 1 - \alpha/2$.)
For the answer to your second question, yes, we can calculate ${\rm E}[|\hat p - p|]$, since $\hat p = X/n$, where $X \sim {\rm Binomial}(n,p)$. I don't know the result off the top of my head.
Addendum. It would seem that the sum $$\sum_{x=0}^n \left|\frac{x}{n} - p\right| \binom{n}{x} p^x (1-p)^{n-x}$$ doesn't have a nice closed form. However, the squared error of course does; it is simply the variance: $${\rm E}[(\hat p - p)^2] = \frac{p(1-p)}{n}.$$
I assume you've got the gist of the idea behind the method from the comments above.
Here's the breakdown of the code:
- Pick a random vector (here a point in 3D) uniformly distributed over a box that contains the volume you'd like to sample. It doesn't have to be a snug fit, the box must simply be bigger.
- If the vector happens to be inside your the region, you'd like to find the volume of, let a counter variable tick one up, e.g. by
k = k+1. - Do this a large number of times, say $N$. Now $$\frac{k}{N}\approx\frac{\text{volume of region}}{\text{volume of box}},$$ and since the volume of the box is easy to calculate, you can find the desired volume. If $N\rightarrow \infty,$ the approximation will become an equality.
If you're having trouble implementing this in some specific software (like Python or Matlab), please ask over at Stack Overflow.
Best Answer
The comments are correct that there are various paths and that it would be helpful if you could give more information. However, here are some generic investigations that I have found useful. They might not be the end of your 'validation', but they may raise issues that point towards the end.
(1) If you have $m = 10^6$ independent iterations in a Monte Carlo simulation of a mean or proportion, then it is reasonable to assume that mean or proportion is nearly normal. So find the relevant SD and use $2SD/\sqrt{m}$ to give a serviceable estimate of the margin simulation error.
(2) If you have questions about independence (as in Markov Chain Monte Carlo, for example) then make an ACF plot for lags out to about 40, and see how long it takes for dependence to decay. Then 'thin' the results; for instance if ACF for lag 10 and beyond is essentially 0, then use every 10th observation. Then adjust m in step (1) accordingly.
(3) If you have questions about the stability of the simulation, then look at 'traces' of running averages (cumulative sums up to n divided by n) against n. They may be very rough-looking as the simulation begins, but they should be quite smooth near the end.
(4) In the case of dependence there is always the risk that the simulation is 'getting stuck' for periods of time (or forever). A 'history plot' of the observations against time will help you to see any serious problems along these lines.
(5) If you are generating several quantities that ought to be stochastically or functionally independent, make scatter plots of pairs of such quantities (along with Pearson and Spearman correlations) to see if there are unexpected relationships. If there are many such quantities, you might try making 'matrix plots' of 'pairs' of variables.
(6) Make histograms of individual simulated quantities. Don't just look at means (or medians) and standard deviations (or interquartile ranges). A histogram of a random quantity should have an 'envelope' or 'density estimator' that is nearly the same as the distribution. If you see unexpected evidence of strong multiple modes, you should try to figure out why.
(7) Sometimes an easily computed by-product of a simulation is a quantity that is has a known value or distribution. Compute them even if they are not part of the reason for doing the simulation. If they do not have the anticipated values or approximate the anticipated distributions, that is a sign of trouble. It is usually better to compare empirical cumulative distribution functions (ECDFs) with known CDFs of distributions than to compare histograms with known PDFs (no loss of acuity due to binning with ECDFs).
I doubt that all of these suggestions will be applicable to any one simulation, but give serious consideration to these and other numerical and graphical descriptions of the output of your simulation. You may find unexpected glitches in the simulation program that are not hard to fix. You may also see that everything is working wonderfully well, and then you will feel much better about the usefulness of your simulation.
Most of the words I have put in 'single quotes' have some chance of fetching useful information in an Internet search if you need further information. Also, when you know a little more about some of these methods and about how your simulation works, you might try posting more specific and technical follow-up questions on our sister 'statistics' site.