Empirical distribution function

probabilityprobability distributionsprobability theory

I am stuck on a problem which seems to simple, meaning there is probably more to it than I am thinking.

Let ${\xi_1,…,\xi_n}$ be independent and identically distributed over ${(\Omega, \mathcal{A}, \mathbb{P})}$, with the continuous distribution function ${F}$. Then $${F_n:= [0,1] \ni t \rightarrow F_n(t) := \frac{1}{n} \sum_{i=1}^n \chi(\xi_i \leq t)}$$
is the empirical distribution function, ${\chi}$ being the indicator function.

1) Find ${nF_n(\xi_i)}$ and ${F_{n}^{-1}(\frac{i}{n})}$ for ${1 \leq i \leq n}$

I doubt that I am right about this, but i simply wrote $${n*F_n(\xi_i) = n*\frac{1}{n} * \sum_{j=1}^n \chi(\xi_j \leq \xi_i) = \sum_{j=1}^n \chi(\xi_j \leq \xi_i)}$$

And my approach for ${F_{n}^{-1}(\frac{i}{n})}$ would be the same, just using $${F_n^{-1}(u):= inf\{ x \in X: F_n(x) \geq u\}}$$

Is my approach correct? Because I did not use the continuous function ${F}$ at all and does not really seem like a solution.

2) Determine the distribution of ${F_n^{-1}(\eta)}$, with ${\eta}$ being equally distributed over ${[0,1]}$

I think here it is a direct conclusion of Skorochod's theorem that ${F_n^{-1}}$ in this case must have the distribution ${F_n}$, although I do not know how to show this other than just to state "implied by Skorochod".

Any help will be appreciated!

Best Answer

For every $i\in\{1,\dots,n\}$ there is a $k\in\{1,\dots,n\}$ such that $\xi_i=\xi_{(k)}$ where $\xi_{(k)}$ denotes the $k$-th order statistic.

Observe that - if $i\neq j$ - we have $\chi(\xi_i\leq\xi_j)\stackrel{a.s.}{=}\chi(\xi_i<\xi_j)$ because $F$ is a continuous distribution.

Actually the continuity of $F$ allows you to assume that: $$\xi_{(1)}<\xi_{(2)}<\cdots<\xi_{(n)}\tag1$$where $<$ replaces $\leq$.

Based on this it is not difficult to find that $nF_n(\xi_i)=k$ where $k$ is the integer that satisfies $\xi_i=\xi_{(k)}$.

Your expression for $F_n^{-1}(u)$ is okay and again applying $(1)$ we find: $$F^{-1}_n\left(\frac{i}{n}\right)=\xi_{(i)}$$

You are correct.

In general if $F$ is a CDF and $\Phi:(0,1)\to\mathbb R$ is defined by: $$\Phi(u)=\inf(\{x\in\mathbb R\mid F(x)\geq u\}$$then it can be deduced that: $$u\leq F(x)\iff \Phi(u)\leq x$$

Based on that we find for $\eta$ uniformly distributed on $(0,1)$:$$P(\Phi(\eta)\leq x)=P(\eta\leq F(x))=F(x)$$

Related Solutions

[Math] Empirical Distribution Function Understanding

Let us denote our probability space by $(\Omega,\mathcal{F},P)$ and let $X_1,X_2,\ldots,X_n$ be a sequence of i.i.d. random variables defined on $\Omega$.

You're correct that $\{X_i\leq x\}$ is shorthand notation for $\{\omega\in\Omega\mid X_i(\omega)\leq x\}$ which is a subset of $\Omega$ that belongs to $\mathcal{F}$ (since $X_i$ is a random variable). Futhermore, $I(X_i\leq x)$ is the indicator function for the set $\{X_i\leq x\}\subseteq\Omega$ and by definition it is a function defined on $\Omega$ (in fact it is a random variable since the set belongs to $\mathcal{F}$): $$ \begin{align} I(X_i\leq x)(\omega)&= \begin{cases} 1,\quad \text{if }\omega\in \{X_i\leq x\},\\ 0,\quad \text{otherwise}. \end{cases} \\ &= \begin{cases} 1,\quad\text{if }X_i(\omega)\leq x,\\ 0,\quad\text{otherwise}. \end{cases} \end{align} $$

Therefore, $\frac1n \sum_{i=1}^n I(X_i\leq x)$ is also a random variable for each fixed $n$.

A sample in this connection just denotes a sequence of i.i.d. random variables $X_1,\ldots,X_n$. An outcome of this sample corresponds to a fixed $\omega$, and $X_1(\omega),\ldots,X_n(\omega)$ would be an outcome or observation of the sample $X_1,\ldots,X_n$.

The empirical distribution function $F_n(x)=\frac1n \sum_{i=1}^n I(X_i\leq x)$ is indeed a random variable, and we can evaluate it in the following way: $$ (F_n(x))(\omega)=\frac1n\sum_{i=1}^n I(X_i(\omega)\leq x), $$ i.e. for a fixed outcome $\omega\in\Omega$, $(F_n(x))(\omega)$ is the number of observations that are less than $x$ divided by $n$ based on the outcome $X_1(\omega),X_2(\omega),\ldots,X_n(\omega)$.

Now suppose we have an infinite sample of i.i.d. variables $X_1,X_2,\ldots$. Then by the law of large numbers one has that for every fixed $x$, the random variables $F_1(x), F_2(x),F_3(x)$ converges almost surely to the true CDF $F$: $$ F_n(x)\to F(x)\;\;\text{almost surely as } n\to\infty. $$

[Math] Definition and use of Empirical Cumulative Distribution Function (ECDF)

Sometimes one says that a histogram based on a large sample size gives a good idea about the shape of the population density function. (But information is lost in binning, and a modern 'density estimator' usually works better.)

In somewhat the same way an empirical cumulative distribution function (ECDF) of a large sample is a good estimator of the population CDF.

The following R program samples 3000 observations from $Gamma(5, 1)$ to illustrate @Clement C's comment. The figure below shows the histogram (at left) along with the known population density (dotted) and a density estimator. At right, the CDF (thin light green) is superimposed on the ECDF (heavy black) of the sample. A larger sample would show better fit, but perhaps too good to see distinctions between population and sample curves.

 x = rgamma(3000, 5, 1)   # generate random sample
 par(mfrow=c(1,2))        # two panels in one graph
   hist(x, prob=T, col="wheat")
     lines(density(x), lwd=2, col="blue")  # density estimator
     curve(dgamma(x, 5, 1), lty="dotted", lwd=2, col="red", add=T)
   plot.ecdf(x)           # empirical CDF
     curve(pgamma(x, 5, 1), col="green", add=T)  # pop CDF
 par(mfrow=c(1,1))        # returns to default single panel

If you have access to R, you can try other population distributions and sample sizes. The same program as above, except with a sample of size $n = 100$ was used to produce the figure below. Roughly speaking, the ECDF gives a better estimate of the CDF than a histogram gives of the PDF. A 'nonparametric bootstrap' procedure uses the sample ECDF in place of the unknown population CDF.

Best Answer

Related Solutions

[Math] Empirical Distribution Function Understanding

[Math] Definition and use of Empirical Cumulative Distribution Function (ECDF)

Related Question