Minimize Chernoff Bound Exponential Distribution

exponential distributionmoment-generating-functions

For the distribution $f_X(x)=\lambda e^{-\lambda x} \ \ x =[0,\infty]$ I am trying to understand how to utilize the Chernoff Bound. The Chernoff bound is:
$P(X>x) \leq g_X(r)e^{-rx}$ where $g_X(r)$ is the moment generating function for the distribution. I have the moment generating function as $\frac{\lambda}{\lambda – r}$. This makes the Chernoff bound $P(X>x) \leq \frac{\lambda}{\lambda – r}e^{-rx}$.

The question is asking which value of r returns the tightest bound ie minimize the Chernoff bound over all possible values of r. From graphing various combinations of lambda and r it seems that have an r value as close to lambda as possible but I am not sure. Having an r value as close to lambda as possible results in steep exponential. However, having a r value of 1 results in a function that starts closer to zero but takes longer to reach zero. What does "tighter" bound mean, and how does having a tighter bound help with using the Chernoff bound?

The following question then asks to compare the Chernoff bound with the Markov bound with $\lambda = 1$. I am further confused because this would mean the only value of r allowed is 0. Have I calculated the Chernoff bound incorrectly?

Best Answer

There are several common notions of tightness of bounds, below is perhaps the simplest one.

Denote the Chernoff bound as $B(x) \equiv \frac{ \lambda }{ \lambda - r} e^{- rx}$ for the exponential function, which tail probability (complement CDF) is $P(X > x) = 1 - F_X(x) = e^{-\lambda x}$.

The error $B(x) - e^{-\lambda x}$ is readily shown to be always positive over $x \in \mathbb{R}^{+}$ for all $0 < r < \lambda$.

Consider the total error over $\mathbb{R}^{+}$, which is an routine integral: \begin{align} \Delta(r) \equiv \int_{ x = 0}^{\infty} \left( \frac{ \lambda }{ \lambda - r} e^{- r x} - e^{-\lambda x} \right)\,\mathrm{d}x = \frac{ \lambda}{ r\, (\lambda - r)} - \frac1{\lambda} \end{align} Note that the relative error $(B(x) - e^{-\lambda x})/e^{-\lambda x}$ doesn't converge as is. You may consider various transformation to make it work if the notion of relative error is really important to you.

Moving on. One can already guess that the minimum of $\Delta(r)$ occurs at $r_0 = \lambda / 2$ in the middle because the function is mirror symmetric (exchange $r \longleftrightarrow \lambda-r$). Anyway, checking that with the first and second derivative of $\Delta(r)$ is routine.

As $r$ varies, the curve $B(x)$ may get closer or further to $P(X > x)$ point-wise at any given $x$, depending on the relation between $r$ and $\lambda$. As a whole, one may consider the $r = r_0 = \lambda / 2$, which minimizes $\Delta(r)$ as defined above, as producing the "tightest" bound.

Sometimes, depending on the applications, one might care only about a specific interval of $a < x < b$. One then optimizes $r$ by minimizing the corresponding error over that interval only. It will be a different $r_0$ for different situation.

BTW, here the error over $\mathbb{R}^{+}$ is minimized to $\min \{\Delta(r)\} = \Delta(\frac{\lambda}2) = 3/\lambda$.


If your so-called Markov bound refers to the common $P( X > x) < \frac{ E[X] }x$, then it is just $B_M(x) = \frac1{\lambda x}$.

Note that the Markov inequality is very loose and the corresponding error doesn't converge, as in it's basically $\int \frac1{x} dx$ that blows up.

The Markov bound as such is a simple function, and one can certainly compare the functional forms of the Chernoff bound $B(x)$ with $B_M(x)$, either in general, or for given $\lambda = 1$ and discuss the numerical details at specific points of $x$.

I'm afraid not much (or too much) can be said without further context.

Related Question