It's been awhile since I've studied QM, but I recall it often being stated that classical mechanics was the limit of quantum mechanics as $\hbar \to 0$. Yes, the Hamiltonian approach to mechanics is often used in both. The Poisson bracket of classical Hamiltonian mechanics has it's quantum mechanical analog in the quantum mechanical commutator. IIRC, this is one way to quantize classical theories. For classical or quantum field theory, one typically works in the Lagrangian framework, where one tries to minimize some action.
Theoretical physics is the field that develops theories about how nature operates. It is fundamentally physics, in that the ultimate goal is to describe reality. It is informed by experiment, and at the same time it extends the results of experiments, making predictions about what has not been physically tested. This is accomplished using the language of mathematics, and often the demands of theoretical physicists force mathematicians to extend this language in new directions, but it is not concerned with developing the language of math. Theoretical physicists are, among other things, physicists who are very well-versed in math (which is not to say other physicists are not - please don't hurt me).
Mathematical physics, on the other hand, is a branch of mathematics. It explores relations between abstract concepts, proves certain results contingent upon certain hypotheses, and establishes an interlinked set of tools that can be used to study anything that happens to match the relations and hypotheses on hand. This branch in particular is motivated by the theories used in physics. It may seek to prove certain truths that were simply assumed by physicists, or carefully delineate the conditions under which certain theories hold, or even provide generally applicable tools to physicists, who can in turn apply them to nature. Mathematical physicists are mathematicians who are intrigued/inspired by physics.
One could say that mathematical physics is concerned with the internal, logical consistency of physical theories, while theoretical physics is concerned with finding the right model to describe the world around us. Very roughly, one might diagram these things as shown below.
$$ \text{Mathematical physics} \Longleftrightarrow \text{Theoretical physics} \Longleftrightarrow \text{Experimental physics} $$
Best Answer
My understanding of the usage of these terms is obviously not perfect, but I can still try. I have to admit beforehand that I have never encountered the term ‘rational mechanics’, maybe someone more knowledgeable can expand on this.
Theoretical Mechanics
This is a term used to differentiate between experimental mechanics (bouncing little balls off each other) and theoretical mechanics (trying to derive equations how little balls bounce off each other). As such, it encompasses classical mechanics, analytical mechanics and rational mechanics.
Classical Mechanics
Classical Mechanics is used in two different contexts: First, it is the antonym to Quantum Mechanics. As such, we usually assume a strictly deterministic world ruled by certain differential equations (in one of the many formulations of classical mechanics, see below), as opposed to a quantum mechanical view where probability densities evolve according to the Schrödinger/Heisenberg equation ($i \hbar \partial_t \Psi = H \Psi$ or $\dot A = \frac{i}{\hbar}[ H , A ] + \partial_t A$).
On the other hand, the term ‘classical mechanics‘ is sometimes used to describe Newtonian Mechanics as opposed to the later developments by Euler, Lagrange, Hamilton and Jacobi. Newtonian mechanics rely on the equation $ F = \dot p = m a $ (assuming $\dot m = 0$) to describe the movement of a point particle of mass m. Since $a = \ddot x$, they usually require two integrations to solve for the trajectory of the particle.
Analytical Mechanics
Analytical Mechanics form the ‘other’ branch of classical mechanics and build upon Newtonian mechanics. It can be divided into three major steps: Lagrangian Mechanics, Hamiltonian Mechanics and mechanics based on the Hamilton-Jacobi equation.
Lagrangian Mechanics
Lagrangian Mechanics starts off with the definition of the Lagrangian $\mathcal{L}$ which acts as a measure for the difference between kinetic and potential energy. It is a function of the coordinates and velocities of all particles:
$$\mathcal{L}(q_1,q_2,\ldots,q_N,\dot q_1,\dot q_2,\ldots,\dot q_N,t) = T - U$$
Lagrange then postulated that the actual trajectories of the particles between time $t_1$ and time $t_2$ are these that minimise the action $S$ as defined by
$$ S = \int_{t_1}^{t_2} \mathcal{L} dt $$
which leads to a variation problem: Find $\{q_i,\dot q_i\}$ such that
$$ \delta S = \int_{t_1}^{t_2} \delta \mathcal{L} dt = 0$$
where $\delta X$ describes the variation of $X$ by its arguments (coordinates and velocities, in our case). As it happens, there is an equation that describes when a given quantity fulfills this requirement, namely the Euler-Lagrange equations. These are:
$$ \partial_{q_i} \mathcal{L} - \frac{\mathrm{d}}{\mathrm{d}t} \partial_{\dot q_i} \mathcal{L} = 0 $$
where $\partial_x = \frac{\partial}{\partial x}$ and I dropped the argumets of $\mathcal{L}$. Note that these equations are of first order in $\{q_i,\dot q_i\}$, as opposed to Newton’s $F = \ddot p$. Furthermore, note that I used $q_i$ to denote the coordinate(s) of the $i$-th particle rather than $x_i$: This is because Lagrangian mechanics makes it very easy to implement generalised coordinates. This is best shown by example:
Assume (in two dimensions) that you have a bolt of length $l$ fixed at the origin $(0,0)$ and a mass $m$ at the other end of the string. Furthermore assume that the bolt has always the same length. To then describe the movement of the mass using the standard coordinates, we need to introduce $y$ and $x$ and integrate each of them and do all sorts of ugly things and, most importantly, always have to take care that $y^2 + x^2 = l^2$. However, we notice that there is only one degree of freedom: the angle. By then introducing a generalised coordinate $q$, we can implement the requirement $y^2 + x^2 = l^2$ by simply not admitting any other coordinates. We set
$$ x = l \cos(q) \quad y = l \sin(q) $$
and can be sure that the bolt always has the same length. Assuming a constant gravitational potential (i.e. potential energy $m \cdot g \cdot x$), we can write
$$ \mathcal{L}(q,\dot q,t) = \frac{1}{2m} l \dot q^2 - m \cdot g \cdot l \cos(q) $$
and hence
$$ m \cdot g \cdot l \sin(q) - \frac{\textrm{d}}{\textrm{d}t} \frac{1}{m} l \dot q = 0 .$$
You might notice that there’s still a $\ddot q$ hidden there. That’s where Hamiltonian Mechanics comes in.
Hamiltonian Mechanics
Hamilton noticed that Lagrangian mechanics is still basically Newtonian mechanics with a nicer dress, but by applying a Legendre transformation to $\mathcal{L}$, we can actually get rid off $\dot q$ (and therefore $\ddot q$).
To this end, we introduce the ‘canonically conjugated momentum‘ $p_j = \partial_{\dot q_j} \mathcal{L}$ and the Hamiltonian $\mathcal{H}$ which is a function of $q$, $p$ (and, rarely, $t$), defined by:
$$\mathcal{H}(q_1,q_2,\ldots,q_N,p_1,p_2,\ldots,p_N,t) = \sum_i \dot q_i p_i - \mathcal{L} \quad = T + U.$$
You might want to verify that $\mathcal{H}$ does not depend on $\dot q_i$, but only on the canonically conjugated coordinates and their momentums $\{ q_i , p_i \}$. We can then rewrite the Euler-Lagrange equations as follows:
$$ \dot q_i = \partial_{p_i} \mathcal{H} \quad ; \quad \dot p_i = - \partial_{q_i} \mathcal{H} \quad .$$
You can memorise these by setting $H(q,p,t) = T(p) + U(q)$, the second term then becomes ‘something like’ $\nabla U = -F = - \dot p_i$.
These equations are still asymmetric (hence the rule above), but by introducing Poisson brackets:
$$ \{ A , B \} = \sum_i [ \partial_{q_i} A \partial_{p_i} B - \partial_{p_i} A \partial_{q_i} B ] $$
we can actually fix that. Observe:
$$ \dot q_i = \{ q_i , H \} \quad ; \quad \dot p_i = \{ p_i , H \} $$
since $\partial_{q_i} p_j = 0 \quad \forall i,j$.
Furthermore, we can now leap to quantum mechanics with relative ease: Simply add a ‘hat’ to $\mathcal{H}$ and replace $\{ \cdot , \cdot \}$ by $-\frac{i}{\hbar} [ \cdot , \cdot ]$, where $[ A , B ] = AB - BA$ denotes the commutator :)
If you feel particularly nasty, look up the Hamilton-Jacobi equation, it is really not nice (or the most beautiful thing ever seen, depending on your POV).