How to Bound the Probability That a Point Belongs to a Set

coveringexpectationpr.probabilityupper bounds

Let $(a_k)_{k \geq 1}$ be random variables taking values on a finite subset $B$. Assume that
$$
(1) \quad \Pr\Big (\lim_{n\rightarrow +\infty}d(\frac{1}{n}\sum_{k=1}^n 1_{[a_k = b]}, [v_\ell(b,\theta_0),v_u(b,\theta_0)])= 0\Big)=1 \quad \forall b\in B
$$

where $d$ stays for distance (i.e., absolute value difference); $v_\ell$ and $v_u$ are real-valued functions of $b\in B$ and $\theta \in \mathbb{R}$; $\theta_0\in \mathbb{R}$ is a specific value of $\theta$.

Define
$$
\Theta_n\equiv \Big\{\theta \in \mathbb{R}: \frac{1}{n}\sum_{k=1}^n 1_{[a_k = b]}\in [v_\ell(b,\theta),v_u(b,\theta)] \forall b \in B\Big\}
$$

Question: Is it possible to find an upper for $\Pr(\theta_0\in \Theta_n)$?

My attempt so far:
We have that $E\Big(\frac{1}{n}\sum_{k=1}^n 1_{[a_k = b]}\Big)\leq v_u(b,\theta_0)$ (see here). By Markov inequality
$$
\Pr \Big(\frac{1}{n}\sum_{k=1}^n 1_{[a_k = b]} \geq r\Big)\leq \frac{v_u(b,\theta_0)}{r}
$$

for each $r>0$. From here?


As advised in the comment below, the problem as posed only provides trivial bounds. Therefore, in what follows, I add some additional structure. In particular, I show that
$$
(2) \quad \Pr(a_k=b | a_{1},\dots, a_{k-1}) \in \big[\nu_{\ell}(b, \theta_0), \nu_u(b,\theta_0)\big] \quad \text{ for all } k, b
$$

is a sufficient condition for (1), in the hope that this can suggest additional relevant details.

Proof of (1):
From (2), we have that
$$
\frac{1}{n}\sum_{k=1}^n \Pr(a_k=b | a_{1},\dots, a_{k-1})\in \big[\nu_{\ell}(b, \theta_0), \nu_u(b,\theta_0)\big] \text{ for all } n
$$

Let
$$
S_n(b)\equiv\sum_{k=1}^n \frac{1}{k} \left({1}_{[a_k=b]}- \Pr(a_k=b| a_1, \dots, a_{k-1})\right).
$$

Observe that the sequence $(S_n(b))_{n\geq 1}$ is an $L^2$-bounded
martingale. Therefore, $S_n(b)$ converges almost surely as $n\to +\infty$. Further, $\frac{1}{n}\sum_{k=1}^{n-1} S_k(b)$ converges to the same limit as $S_n(b)$.

Let
$$
Q_n(b)\equiv \sum_{k=1}^n \left(1_{[a_k=b]}- \Pr(a_k=b| a_1, \dots, a_{k-1}\right).
$$

Observe that
$$
Q_n(b) = n S_n(b)-\sum_{k=1}^{n-1} S_{k}(b).
$$

By the convergence of $S_n(b)$ and $\frac{1}{n}\sum_{k=1}^{n-1} S_k(b)$ to the same limit, $\frac{1}{n}{Q_n(b)}$ converges almost surely to 0 as $n\to +\infty$.

Hence,
$$
\frac{1}{n}\sum_{k=1}^n 1_{[a_k=b]}-\frac{1}{n}\sum_{k=1}^n \Pr(a_k=b| a_1, \dots, a_{k-1})\overset{a.s.}{\to} 0 \text{ as $n\to +\infty$.}
$$

Therefore, $\frac{1}{n}\sum_{k=1}^n 1_{[a_k=b]}$ and $\frac{1}{n}\sum_{k=1}^n \Pr(a_k=b| a_1, \dots, a_{k-1})$ have the same limit points which belong to $ \big[\nu_{\ell}(b, \theta_0), \nu_u(b,\theta_0)\big] $ and (1) holds.

Best Answer

As I wrote before in a comment, the only upper bound on $\Pr(\theta_0\in\Theta_n)$ under these very general conditions is the trivial bound $1$.

Indeed, suppose that for some $b_*\in B$ \begin{equation} \Pr(a_1=a_2=\cdots=b_*)=1 \end{equation} and for all $b\in B$ \begin{equation} v_\ell(b,\theta_0)=1(b=b_*)=v_u(b,\theta_0). \end{equation} Then $\Pr(a_k=b)=v_\ell(b,\theta_0)=v_u(b,\theta_0)$ for all natural $k$ and all $b\in B$. So, condition (1) holds and $\Pr(\theta_0\in\Theta_n)=1$ for all natural $n$.

Related Question