Neither of them holds, in general, for stochastic integrals.
The trouble already starts if you consider measures which need not to be non-negative, i.e. signed measures. For a signed measure $\mu: (\Omega,\mathcal{A}) \to \mathbb{R}$ we cannot expect that the triangle inequality
$$\left| \int f(x) \, \mu(dx) \right| \leq \int |f(x)| \, \mu(dx) \tag{1}$$ holds. To see this, just consider the case that $f$ is an elementary function, i.e. $f$ is of the form
$$f(x) = \sum_{j=1}^n c_j 1_{A_j}(x).$$
Then $(1)$ reads
$$\left| \sum_{j=1}^n c_j \mu(A_j) \right| \leq \sum_{j=1}^n |c_j| \mu(A_j).$$
Note the right-hand side does not even need to be non-negative (but the left-hand side is), so this doesn't make any sense. With the same reasoning, we find that
$$\left| \int f(x) \, \mu(dx) \right| \leq \|f\|_{L^{\infty}} \mu(\Omega) \tag{2}$$
does, in general, not hold true for signed measures. However, one can show that
$$|\mu|(A) := \sup \left\{ \sum_{n \in \mathbb{N}} |\mu(A_n)|; A_n \in \mathcal{A} \, \text{disjoint}, \bigcup_{n \in \mathbb{N}} \subseteq A \right\}$$
defines a non-negative measure, the so-called total variation norm, and that
$$\left| \int f \, d\mu \right| \leq \|f\|_{\infty} |\mu|(\Omega)$$
and
$$\left| \int f \, d\mu \right| \leq \int |f| \, d|\mu|.$$
These two are the natural generalizations of $(1)$ and $(2)$ for signed measures.
Since stochastic integrals are "randomized" signed measures, the situation becomes even more complicated. For example if $(M_t)_{t \geq 0}$ is a Brownian motion, then the stochastic integral
$$\int_0^t H_s \, dM_s$$
is not a pointwise integral and this means that we cannot simply use the above considerations for fixed $\omega$. Additionally, there is the trouble that the Brownian motion has infinite total variation, so, as far as I can see, there is no chance to get such inequalities for stochastic integrals with respect to Brownian motion.
Very important inequalities for stochastic integrals (with respect to martingales) are e.g.
- Doob's inequality
- the Burkholder-Davis-Gundy inequality
but they don't provide any pointwise estimates.
The only exception I can think of are processes with bounded variation. In this case, we can define the stochastic integrals as a Riemann-Stieltjes integral and obtain similar estimates as for signed measures. This works in particular for processes with non-decreasing sample paths, e.g. subordinators.
Best Answer
While I agree with the comments that this is an art rather than a science, there are patterns, and one important one (at least for me) came from The Cauchy-Schwarz Master Class (J. Michael Steele).
You typically get a "good deal" from inequalities when you're working in a region where the inequality is sharp.
As an (obvious) example, say you're working with the approximation $e^x \geq 1 + x$. Obviously this approximation is terrible most of the time (proof by picture below)
But notice this inequality is sharp, (i.e. $e^x = 1+x$) when $x = 0$. This tells us (basically by continuity) that this equality is a good deal when $x \approx 0$. The graph above shows this.
This is a good example, because we know from calculus exactly how good a deal this approximation is, and why it becomes a better and better deal as $x \to 0$. Thankfully, this same heuristic can be used in many important inequalities.
Since you mention the AM-GM inequality, let's ask when is AM-GM sharp?
One can show (and indeed you should) that $(a_1 a_2 \cdots a_n)^{1/n} = \frac{a_1 + a_2 + \cdots + a_n}{n}$ iff the $a_i$ are all the same.
This gives us a clue as to when we should use the AM-GM inequality: we get a good deal whenever the $a_i$ are approximately equal! Let's try to use this to prove something fun: Carleman's Inequality.
$$\sum \limits_{k=1}^\infty (a_1 a_2 \cdots a_k)^{1/k} \leq e \sum \limits_{k=1}^\infty a_k$$
The left side of this is crying out for the AM-GM inequality! You might suspect that hitting the left side with AM-GM will give rise to something with the desired sum, plus some extra $\frac{1}{n}$ terms which we can massage into $(1 + \frac{1}{n})^n$ to get our factor of $e$. Unfortunately...
$$ \sum_{k=1}^n (a_1 a_2 \cdots a_k)^{1/k} \leq \sum_{k=1}^n \frac{1}{k} \sum_{j=1}^k a_j = \sum_{j=1}^n a_j \sum_{k=j}^n \frac{1}{k} $$
This sum will diverge in the limit, and it seems we're out of luck. The problem, as you may have guessed from the fact that I'm using this as an example, is that we are applying the AM-GM inequality where it is weak. If we can reframe the problem in a way that guarantees the $a_k$ are all roughly the same, then we can get a better bargain when using it.
Let's fix this problem with the least possible effort - we'll introduce a new sequence $c_k$ such that $c_1 a_1 \approx c_2 a_2 \approx c_3 a_3 \approx \cdots$. If we repeat our calculation with these new fudge factors, we can figure out later what exactly they should be to make the inequality work out.
$$ \begin{align*} \sum \limits_{k=1}^\infty (a_1 a_2 \cdots a_k)^{1/k} &= \sum \limits_{k=1}^\infty \frac{(a_1 c_1 a_2 c_2 \cdots a_k c_k)^{1/k}}{(c_1 c_2 \cdots c_k)^{1/k}}\\ &\leq \sum \limits_{k=1}^\infty \frac{a_1c_1 + a_2c_2 + \cdots + a_k c_k}{k(c_1 c_2 \cdots c_k)^{1/k}}\\ &= \sum \limits_{k=1}^\infty a_k c_k \sum \limits_{j=k}^\infty \frac{1}{j(c_1 c_2 \cdots c_j)^{1/j}} \end{align*} $$
Ok, at some point we have to actually do work, and this is where that happens. We want to make $$s_k = c_k \sum \limits_{j=k}^\infty \frac{1}{j(c_1 c_2 \cdots c_j)^{1/j}}$$ a convergent sequence. Preferably one which is bounded above by $e$, so that we can recover the desired inequality. We again take the easiest path to making this sum converge - let's force it to telescope. Recall
$$\sum \limits_{j=k}^\infty \frac{1}{j(j+1)} = \sum \limits_{j=k}^\infty \frac{1}{j} - \frac{1}{j+1} = \frac{1}{k} $$
Our sequence $c_k$ is still undefined. If we define it so that $(c_1 c_2 \cdots c_j)^{1/j} = j+1$, that forces
$$ s_k = c_k \sum \limits_{j=k}^\infty \frac{1}{j(c_1 c_2 \cdots c_j)^{1/j}} = c_k \sum \limits_{j=k}^\infty \frac{1}{j(j+1)} = \frac{c_k}{k} $$
But what is $c_j$? Notice $(c_1 c_2 \cdots c_{j-1}) = j^{j-1}$ and $(c_1 c_2 \cdots c_j) = (j+1)^j$. Thus
$$c_j = \frac{(j+1)^j}{j^{j-1}} = j \left ( 1 + \frac{1}{j} \right )^j$$
Now we see the light at the end of the tunnel!
$$ \begin{align*} \sum \limits_{k=1}^\infty (a_1 a_2 \cdots a_k)^{1/k} &\leq \sum \limits_{k=1}^\infty a_k s_k\\ &= \sum \limits_{k=1}^\infty a_k \frac{c_k}{k}\\ &= \sum \limits_{k=1}^\infty a_k \left ( 1 + \frac{1}{k} \right )^k\\ &\leq e \sum \limits_{k=1}^\infty a_k \end{align*} $$
This example was fairly long, but hopefully shows how we can ensure we get a "good deal" from our inequalities just by ensuring we use them where they are sharp. This example was (almost verbatim) taken from the book I mentioned at the top of the post, The Cauchy-Schwarz Master Class. It is absolutely worth reading, and changed the way that I (a dyed-in-the-wool algebraist) approach analysis and inequalities. It is packed full of tricks and tips for getting the most out of your inequalities, and does a good job explaining how and when to apply them at their full power. I found it useful, and I hope others do too!
I hope this helps ^_^