You draw a picture in the y-z plane, and you rotate it by 45 degrees and blow up by a factor of $\sqrt{2}$. This gives you a triangle region, where the vertices are determined to be the image of the old vertices under the rotation: (x,x),(a,x),(a,a). The images in the $\beta,\gamma$ plane are $(2x,0),(a+x,a-x),(2a,0)$.
But I understand that you want a formal algorithm, so that you can do it mechanically. First, you make a one-variable transformation, where you trade in z for $\gamma= z-y$. If you really insist on doing the rotation (it's not the right thing), you can then trade in $y$ for $\beta$, using $\beta= 2y + \gamma$.
The answer for the first shift is that the z range is from y to a, so translating z by -y goes from 0 to a-y.
$$\int_0^a dx f(x) \int_x^a dy \int_0^{a-y} d\gamma g(\gamma) $$
To complete the mechanical change of variable, introduce an indicator function of one dimension $\phi(x)$, which is 1 for $x>0$ and 0 for $x<0$. Then you write the integral as follows:
$$ \int_0^a dx f(x) \int_x^a dy \int_0^\infty \phi(a-y-\gamma) g(\gamma) d\gamma$$
Rearrange the order of the y and $\gamma$ integrals and perform the y integral explicitly (using the fact that the indefinite integral of $\phi(x)$ is $x\phi(x)$) to get
$$\int_0^a dx f(x) \int_0^\infty (a-x-\gamma)\phi(a-x-\gamma)) g(\gamma) d\gamma $$
The $\phi$ function now gives you the new domain for $\gamma$
$$ \int_0^a dx f(x) \int_0^{a-x} (a-x-\gamma)g(\gamma)$$
and this is the best form. Using explicit indicator functions ($\phi$) this way is useful in many cases for getting explicit answers in closed form. If you really want to do the transformation of variables the way you said it, you introduce the indicator functions for the region, and then transform the region and find the new domain. This is just the same as drawing the triangle and rotating it.
When integrating a function $f(x,y)$,
$$\int_A^B\int_{C(y)}^{D(y)} f(x,y) dxdy$$
you can rewrite the integration region as
$$ \{(x,y): A\le y \le B , C(y) \le x \le D(y) \}$$
For reasonable functions $C,D$, this is equivalent to
$$ \{ (x,y) : \inf C \le x \le \sup D , D^{-1}(x)<y<C^{-1}(x)\}$$
which gets you
$$\int_{\inf C}^{\sup D} \int_{D^{-1}(x)}^{C^{-1}(x)} f(x,y) dydx$$
Best Answer
A more precise notation is this one
$$\int_{x_{1}}^{x_{2}}u(x)v^{\prime }(x)dx=\left( u(x_{2})v(x_{2})-u(x_{1})v(x_{2})\right) -\int_{x_{1}}^{x_{2}}u^{\prime }(x)v(x)dx$$
which is derived from the derivative rule for the product
$$(u(x)v(x))^{\prime }=u^{\prime }(x)v(x)+u(x)v^{\prime }(x)$$
or
$$u(x)v^{\prime }(x)=(u(x)v(x))^{\prime }-u^{\prime }(x)v(x).$$
So
$$\begin{eqnarray*} \int_{x_{1}}^{x_{2}}u(x)v^{\prime }(x)dx &=&\int_{x_{1}}^{x_{2}}(u(x)v(x))^{\prime }dx-\int_{x_{1}}^{x_{2}}u^{\prime }(x)v(x)dx \\ &=&\left. (u(x)v(x))\right\vert _{x=x_{1}}^{x=x_{2}}-\int_{x_{1}}^{x_{2}}u(x)v(x)dx \\ &=&\left( u(x_{2})v(x_{2})-u(x_{1})v(x_{2})\right) -\int_{x_{1}}^{x_{2}}u^{\prime }(x)v(x)dx. \end{eqnarray*}.$$
If you write $dv=v^{\prime }(x)dx$ and $du=u^{\prime }(x)dx$, you get your formula but with $u,v$ as a function of $x$
$$\int_{v_{1}(x)}^{v_{2}(x)}u(x)dv=\left( u(x_{2})v(x_{2})-u(x_{1})v(x_{2})\right) -\int_{u_{1}(x)}^{u_{2}(x)}v(x)du$$
Example: Assume you want to evaluate $\int_{x_{1}}^{x_{2}}\log xdx=\int_{x_{1}}^{x_{2}}1\cdot \log xdx$. You can choose $v^{\prime }(x)=1$ and $u(x)=\log x$. Then $v(x)=x$ (omitting the constant of integration) and $u^{\prime }(x)=\frac{1}{x}$. Hence
$$\begin{eqnarray*} \int_{x_{1}}^{x_{2}}\log xdx &=&\int_{x_{1}}^{x_{2}}1\cdot \log xdx \\ &=&\left( \log x_{2}\cdot x_{2}-\log x_{1}\cdot x_{1}\right) -\int_{x_{1}}^{x_{2}}\frac{1}{x}\cdot xdx \\ &=&\left( \log x_{2}\cdot x_{2}-\log x_{1}\cdot x_{1}\right) -\int_{x_{1}}^{x_{2}}dx \\ &=&\left( \log x_{2}\cdot x_{2}-\log x_{1}\cdot x_{1}\right) -\left( x_{2}-x_{1}\right) \end{eqnarray*}$$
The same example with your formula:
$$u=\log x,v=x,dv=dx,v=x,du=\frac{1}{x}dx$$
$$u_{2}=\log x_{2},u_{1}=\log x_{1},v_{2}=x_{2},v_{1}=x_{1}$$
$$\begin{eqnarray*} \int_{v_{1}}^{v_{2}}udv &=&\left( u_{2}v_{2}-u_{1}v_{2}\right) -\int_{u_{1}}^{u_{2}}vdu \\ \int_{x_{1}}^{x_{2}}\log xdx &=&\left( \log x_{2}\cdot x_{2}-\log x_{1}\cdot x_{1}\right) -\int_{\log x_{1}}^{\log x_{2}}xdu \\ &=&\left( \log x_{2}\cdot x_{2}-\log x_{1}\cdot x_{1}\right) -\int_{x_{1}}^{x_{2}}x\cdot \frac{1}{x}dx \\ &=&\left( \log x_{2}\cdot x_{2}-\log x_{1}\cdot x_{1}\right) -\left( x_{2}-x_{1}\right). \end{eqnarray*}$$
Note: The limits of integration, although different in terms of $u(x),v(x)$, when expressed in terms of the same variable $x$ of functions $u(x),v(x)$ are the same in both sides.
For a strategy on how to chose the $u$ and $v$ terms see this question.