[Math] Euler-Lagrange equations and Bellman’s principle of optimality

calculus-of-variationsoc.optimization-and-control

One method to optimize the integral
$$\int_{\mathcal T} L(t,x,\dot{x}) \; dt $$
of a functional over a curve is the calculus of variations, which leads to ordinary differential equations: the Euler-Lagrange equation
$$-\frac{d}{dt} L_{\dot{x}}(t,x(t),\dot{x}(t)) + L_x(t,x(t),\dot{x}(t)) =0$$
or its generalization, Pontryagin's maximum principle. An alternative is Bellman's optimality principle, which leads to Hamilton-Jacobi-Bellman partial differential equations. Each of the methods has advantages and disadvantages depending on the application, and there are numerous technical differences between them, but in the cases when both are applicable the answers are broadly similar.

The calculus of variations can also be used to optimize a functional
$$\int_{\mathcal X} L(x,u,p) \; dx $$
integrated over a multidimensional space. The resulting Euler-Lagrange equations $$-\frac{\partial}{\partial x} L_{p}(x,u(x),p(x)) + L_u(x,u(x),p(x))$$ are partial differential equations with the space coordinates as independent variables. Is an alternative approach using value functions, leading to optimality conditions along the lines of Bellman's optimality principle, known?

Best Answer

A. multi-dimensional state, one-dimensional time

Multi-dimensional extensions $x\in\mathbb{R}^n$ of the one-dimensional Hamilton-Jacobi-Bellman equations have been considered in Consistency of a Simple Multidimensional Scheme for Hamilton-Jacobi-Bellman Equations (2005).

We present an approximation scheme for second-order Hamilton–Jacobi–Bellman equations arising in stochastic optimal control. The scheme is based on a Markov chain approximation method to solve the nonlinear partial differential equations that govern the optimization problem. The scheme can be readily implemented in any dimension. The consistency of the scheme is proved, which guarantees its convergence.


B. multi-dimensional state, multi-dimensional time

For extensions where both state and time are multi-dimensional, $x\in\mathbb{R}^n$, $t\in\mathbb{R}_+^m$, see Multitime linear-quadratic regulator problem based on curvilinear integral (2009) (and several more recent papers on the multi-time Bellman principle by Constantin Udriste and co-workers).

We introduce a multitime dynamic programming method based on multitime Hamilton- Jacobi-Bellman PDEs. These PDEs are equivalent to multitime Hamilton PDEs system and the multitime maximum principle. The optimal control is characterized means of a multitime variant of the Riccati PDE that may be viewed as a feedback law.

Related Question