If $$ L \to L' = L +\frac{dF(q,t)}{dt}$$ the corresponding Hamiltonian becomes
$$ H \to H' = H - \frac{\partial F(q,t)}{\partial t} $$ as shown here. Moreover, the canonical momentum becomes $$ p \to P = p + \frac{\partial F}{\partial q} $$ while $$ q \to Q = q $$ as shown here.
These formulas allow us to check the invariance of Hamilton's equations explicitly. Concretely,
\begin{align}
\frac{dq}{dt} &= \frac{\partial H}{\partial p} \notag \\
\end{align}
becomes
\begin{align}
\frac{dQ}{dt} &= \frac{\partial H'}{\partial P} \\
\therefore \quad \frac{dq}{dt} &= \frac{\partial \left(H - \frac{\partial F(q,t)}{\partial t} \right)}{\partial P} \\
\therefore \quad \frac{dq}{dt} &= \frac{\partial H}{\partial P} - \frac{\partial }{\partial P} \left( \frac{\partial F(q,t)}{\partial t} \right) \\
\therefore \quad \frac{dq}{dt} &= \frac{\partial H}{\partial P} \\
\therefore \quad \frac{dq}{dt} &= \frac{\partial H}{\partial p} \frac{\partial P}{\partial p} \\
\therefore \quad \frac{dq}{dt} &= \frac{\partial H}{\partial p} \quad \checkmark
\end{align}
where I used that $F$ does not depend on $P$ and $$\frac{\partial P}{\partial p} =\frac{\partial }{\partial p} \left( p+ \frac{\partial F}{\partial q} \right) = 1. $$
Analogously, we can check Hamilton's second equation:
$$ \frac{dp}{dt}= -\frac{\partial H(q,p,t)}{\partial q} .$$
However, there is a subtlety. After the transformation, we have on the right-hand side $\frac{\partial H'(Q,P,t)}{\partial Q}$. But here we need take into account that $p$ also depends on $q$, since $ p \to P = p + \frac{\partial F(Q,t)}{\partial Q} $. Therefore
\begin{align}
\frac{\partial H'(Q,P,t)}{\partial Q} &= \frac{\partial H'(Q,p + \frac{\partial F}{\partial q} ,t)}{\partial Q} \\
&= \frac{\partial H'(Q,p,t)}{\partial Q} + \frac{\partial H(Q,p,t) }{\partial p} \frac{\partial p}{\partial Q} \\
&= \frac{\partial H(Q,p,t)}{\partial Q} - \frac{\partial^2 F(Q,t)}{\partial Q \partial t} + \frac{\partial H(Q,p,t) }{\partial p} \frac{\partial p}{\partial Q} \\
&= \frac{\partial H(Q,p,t)}{\partial Q} - \frac{\partial^2 F(Q,t)}{\partial Q \partial t} + \dot Q \frac{\partial \left(P- \frac{\partial F}{\partial q}
\right)}{\partial Q} \\
&= \frac{\partial H(Q,p,t)}{\partial Q} - \frac{\partial^2 F(Q,t)}{\partial Q \partial t} - \dot Q \frac{\partial
}{\partial Q} \frac{\partial F}{\partial q} \\
&= \frac{\partial H(Q,p,t)}{\partial Q} - \frac{d}{dt} \frac{\partial F}{\partial Q} \,.
\end{align}
where we used that
$$ \frac{d}{dt} \frac{\partial F}{\partial Q}= \frac{\partial^2 F(Q,t)}{\partial Q \partial t} + \dot Q \frac{\partial
}{\partial Q} \frac{\partial F}{\partial q} . $$
Using this, we can rewrite Hamilton's second equation after the transformation as follows:
\begin{align}
\frac{dP}{dt}&= -\frac{\partial H'(Q,P,t)}{\partial Q} \\
\therefore \quad \frac{d}{dt} \left( p+ \frac{\partial F(q,t)}{\partial q} \right) &= \frac{\partial H(Q,p,t)}{\partial Q} - \frac{d}{dt} \frac{\partial F}{\partial Q} \\
\therefore \quad \frac{dp}{dt} + \frac{d}{dt} \left(\frac{\partial F(q,t)}{\partial q} \right)&= \frac{\partial H(Q,p,t)}{\partial Q} - \frac{d}{dt} \frac{\partial F}{\partial Q} \\
\therefore \quad \frac{dp}{dt} &= -\frac{\partial H}{\partial q} \quad \checkmark
\end{align}
EDIT: The subtlety was also noted here, but unfortunately without an answer and a few years ago there was even a paper which didn't notice it and claimed that Hamilton's equations are not invariant.
When you write this:
$$D_{t_2} S=-x_2^2\csc^2t_2.\tag6$$
You obviously mean total time derivative by $D_{t2}$. Which means:
$D_{t2}S=dS/dt_2=\frac{\partial S}{\partial q}\frac{\partial q}{\partial t_2}+\frac{\partial S}{\partial t_2}\frac{\partial t_2}{\partial t_2}$
In our case: $q=x_2$
and you're missing this part: $\frac{\partial S}{\partial q}\frac{\partial q}{\partial t_2}$, which is not zero. So, we have:
$D_{t_2} S=-x_2^2\csc^2t_2+\frac{\partial S}{\partial x_2}\frac{\partial x_2}{\partial t_2}=-x_2^2\csc^2t_2+2x_2\cot t_2\frac{dx_2}{dt_2}$
Best Answer
Yes, this is e.g considered in Ref. 1. In field theory, the starting point is the off-shell action$^1$
$$\tag{1} I[\phi; t_f,t_i] ~:=~\int_{t_i}^{t_f} \! dt~\int_{\Sigma} d^3x~ {\cal L}(\phi(x,t),\dot{\phi}(x,t), \partial_x \phi(x,t);x,t) , $$
where $t_i$ and $t_f$ denote initial and final times, respectively. We now impose appropriate boundary conditions (B.C.), e.g. Dirichlet B.C.
$$\tag{2} \phi^{\alpha}(x,t_i)~=~\phi^{\alpha}_i(x) \qquad \text{and}\qquad \phi^{\alpha}(x,t_f)~=~\phi^{\alpha}_f(x) . $$
We assume that for given B.C. (2) there exists a unique solution $\phi_{\rm cl}$ to the Euler-Lagrange equations. OP is interested in the (Dirichlet) on-shell action defined as
$$\tag{3} S[\phi_f,t_f; \phi_i,t_i] ~:=~ I[\phi_{\rm cl}; t_f,t_i].$$
Next define (Lagrangian) momentum field
$$\tag{4} \pi_{\alpha}(x,t) ~:=~\frac{\partial {\cal L}(\phi(x,t),\dot{\phi}(x,t), \partial_x \phi(x,t);x,t)}{\partial \dot{\phi}^{\alpha}(x,t)}, $$
and energy
$$\tag{5} h(t)~:=~\int_{\Sigma} d^3x~\left(\sum_{\alpha}\pi_{\alpha}(x,t)\dot{\phi}^{\alpha}(x,t) -{\cal L}(\phi(x,t),\dot{\phi}(x,t), \partial_x \phi(x,t);x,t)\right).$$
Then one may show field-theoretically$^{2}$ that
and
Example: A free field Lagrangian density ${\cal L} = \frac{1}{2}\phi^2$ leads to
$$ \tag{8} S(\phi_f,t_f; \phi_i,t_i) ~=~ \frac{1}{2(t_f-t_i)} \int_{\Sigma} d^3x~(\phi_f(x)-\phi_i(x))^2 .$$
References:
MTW; Section 21.1 and Section 21.2.
L.D. Landau & E.M. Lifshitz, Mechanics, vol. 1, 1976; $\S$ 43.
--
$^1$ For actions in point mechanics, see e.g. this Phys.SE post.
$^2$ For a proof in point mechanics, see e.g. Ref. 2 and my Phys.SE answer here.