[Tex/LaTex] How to properly write this equation in Latex

Question

The equation is basically Bellman equation essential to the Q-learning algorithm. If there's any way to include the notes below (such as "New Q-Value") aswell that'd be even better!

Answer 1

My welcome to TeX.SE and you here have a my proposal done as pure LaTeX. There is a bit of difference from your image because there is much text in a specific part of the formula (Maximum predicted reward, given new state and all possible actions).

Some small suggestions:

1. In LaTeX there is a specific command to indicate the maximum and it is \max that is not written in italics (max) like each letter is a variable. The same discussion for "New", which I wrote as text into a formula \text{New}, not in italics.
2. An important indication: if you change something in the formula that you have inserted as an image, the spaces may change. This can be easily resolved by reducing or increasing the distance with the command \mkern-30mu (for example).

\documentclass[a4paper,12pt]{article}
\usepackage{amssymb,mathtools}
\begin{document}
\[\underbrace{\text{New}Q(s,a)}_{\scriptstyle\text{New Q-Value}}=Q(s,a)+\mkern-34mu\underset{\text{New Q-Value}}{\underset{\Bigl|}{\alpha}}\mkern-30mu[\underbrace{R(s,a)}_{\scriptstyle\text{Reward}}+\mkern-30mu\underset{\text{Discount rate}}{\underset{\Biggl|}{\gamma}}\mkern-75mu\overbrace{\max Q'(s',a')}^{\scriptstyle\substack{\text{Maximum predicted reward, given} \\ \text{new state and all possible actions}}}\mkern-45mu-Q(s,a)]\]
\end{document}