Bare with me, I don't remember every little step, but I hope this derivation helps you.
First remember how a wave travels through a waveguide (dielectric).
$$ E(x,y,z) = E^{0}(x,y)e^{-\gamma z}$$
$$ H(x,y,z) = H^{0}(x,y)e^{-\gamma z}$$
Then consider Ampere's and Faraday's Laws for a source-free region. $$ \triangledown \times H = j\omega\epsilon E $$ $$ \triangledown \times E = -j\omega\mu H $$
This produces 3 equations each (for the x, y and z directions):
$$ 1) \frac{\partial E_{z}}{\partial y} + \gamma E_{y} = -j\omega\mu H_{x}$$
$$ 2) \frac{\partial E_{z}}{\partial x} + \gamma E_{x} = j\omega\mu H_{y}$$
$$ 3) \frac{\partial E_{y}}{\partial x} - \frac{\partial E_{x}}{\partial y} = -j\omega\mu H_{z}$$
$$ 4) \frac{\partial H_{z}}{\partial y} + \gamma H_{y} = j\omega\epsilon E_{x}$$
$$ 5) \frac{\partial H_{z}}{\partial x} + \gamma H_{x} = -j\omega\epsilon E_{y}$$
$$ 6) \frac{\partial H_{y}}{\partial x} - \frac{\partial H_{x}}{\partial y} = j\omega\epsilon E_{z}$$
We can combine (1) and (5) and combine (2) and (4) due to like terms to generate equations for $H_{x}$ and $E_{x}$ which become (7) and (9). We rearrange equations (3) and (6) for $H_{y}$ and $E_{y}$, respectively.
$$ 7) H_{x} = \frac{-\gamma}{h^2} \frac{\partial H_{z}}{\partial x} + \frac{j\omega\epsilon}{h^2} \frac{\partial E_{z}}{\partial y}$$
$$ 8) H_{y} = \frac{-\gamma}{h^2} \frac{\partial H_{z}}{\partial x} - \frac{j\omega\epsilon}{h^2} \frac{\partial E_{z}}{\partial x}$$
$$ 9) E_{x} = \frac{-\gamma}{h^2} \frac{\partial E_{z}}{\partial x} - \frac{j\omega\mu}{h^2} \frac{\partial H_{z}}{\partial y}$$
$$ 10) E_{x} = \frac{-\gamma}{h^2} \frac{\partial E_{z}}{\partial y} + \frac{j\omega\mu}{h^2} \frac{\partial H_{z}}{\partial x}$$
And remember $ h^{2} = \gamma^{2} + \beta^{2}$, where $\beta = \omega \sqrt{\mu\epsilon}$
Transverse components $E_{x}, E_{y}, H_{x}, H_{y}$ are expressed in terms of the longitudinal components $E_{z}, H_{z}$. And we are given three cases:
1) Transverse Electric (TE): $$ E_{z} = 0, H_{z} \neq 0$$
2) Transverse Magnetic (TM): $$ E_{z} \neq 0, H_{z} = 0 $$
3) Transverse Electromagnetic (TEM): $$ E_{z} = H_{z} = 0 $$
Where in the case of TEM modes equations (7) through (10) break down unless $ h = 0 $ meaning:
$$ \gamma^{2} + \beta^{2} = 0 $$
$$ \gamma^{2} = -\beta^{2} $$
$$ \gamma = j\beta = j\omega\sqrt{\mu\epsilon} $$
Now here we need to bring in the Helmholtz equation to solve the partial differential for TE and TM modes: $$ \triangledown^{2} A + k^{2}A = 0 $$
In TE modes, we need $H_{z}$, which is our $A$ in the Helmholtz equation, and our factor $k^{2}$ is $\beta^{2}$.
Substituting: $$ \triangledown^{2} H_{z} + \beta^{2} H_{z} = 0 $$
Expand: $$ \frac{\partial^{2} H_{z}}{\partial x^{2}} + \frac{\partial^{2} H_{z}}{\partial y^{2}} + \frac{\partial^{2} H_{z}}{\partial z^{2}} + \beta^{2} H_{z} = 0 $$
$$ \frac{\partial^{2} H_{z}}{\partial z^{2}} = -\gamma^{2}H_{z}^{0}(x,y)e^{-\gamma z} $$
$$ \frac{\partial^{2} H_{z}^{0}}{\partial x^{2}} + \frac{\partial^{2} H_{z}^{0}}{\partial y^{2}} + (\gamma^{2} + \beta^{2})H_{z}$$ Since $h^{2}$ = $\gamma^{2} + \beta^{2}$ we conclude: $$ \frac{\partial^{2} H_{z}^{0}}{\partial x^{2}} + \frac{\partial^{2} H_{z}^{0}}{\partial y^{2}} + h^{2}H_{z} = 0$$
Repeat the same steps above for TM modes, where we need $E_{z}$
$$ \triangledown^{2} E_{z} + \beta^{2} E_{z} = 0 $$
And therefore: $$ \frac{\partial^{2} E_{z}^{0}}{\partial x^{2}} + \frac{\partial^{2} E_{z}^{0}}{\partial y^{2}} + h^{2}E_{z} = 0$$
Now the reason for there being only two sets of components is due to the fact that the wave propagates along a single direction given the waveguide. The key factor being that Electric and Magnetic Fields are ALWAYS perpendicular to eachother. This is a primary principle that Maxwell discovered. The two always travel together in electromagnetic waves.
For example:
Where in TM modes the electric field is in the direction perpendicular to that of propagation, so ONLY the magnetic field propagates within the waveguide, and vice versa for TE modes. This is why the electric or magnetic components are considered 0 (given that we're assuming z to be the direction of propagation).
So you have two instances for TM and TE waves, where the electric field is zero or the magnetic field is zero - why you have two sets of equations.
This differs in TEM modes where neither propagate in the direction of the waveguide, however at least two conductors are required for any TEM modes to exist.
The square bracket transformation
This is just the application of chain rule. The LHS means a derivative over the primed spacial coordinates while keeping unprimed spacial and time coordinates fixed.
$$\nabla'[ \rho(\mathbf{x'},t')]_{ret} = \left(\sum_i \frac{\partial }{\partial x_i'} \hat{i}\right)[\rho(x_i',x_j',x_k',t')]_{ret}\\$$
But the $\rho$ is a function of primed time coordinate as well. So the gradient operator has to applied using chain rule.
$$
=\left \lbrace \sum_i \left(\frac{\partial x_i'}{\partial x_i'}\frac{\partial }{\partial x_i'} + \frac{\partial x_j'}{\partial x_i'}\frac{\partial }{\partial x_j'} + \frac{\partial x_k'}{\partial x_i'}\frac{\partial }{\partial x_k'} + \frac{\partial t'}{\partial x_i'}\frac{\partial }{\partial t'}\right) \hat{i}\right\rbrace[\rho(x_i',x_j',x_k',t')]_{ret}\\
=\left \lbrace \sum_i \left(\frac{\partial }{\partial x_i'} + 0+ 0 + \frac{\partial t'}{\partial x_i'}\frac{\partial }{\partial t'}\right) \hat{i}\right\rbrace[\rho(x_i',x_j',x_k',t')]_{ret}\\
$$
So the primed spacial derivatives are taken while keeping the primed t constant.
$$
=\left \lbrace \left(\frac{\partial }{\partial x_i'} \hat{i}+\frac{\partial }{\partial x_j'} \hat{j}+\frac{\partial }{\partial x_k'} \hat{k}\right) + \left(\frac{\partial t'}{\partial x_i'} \hat{i}+\frac{\partial t'}{\partial x_j'} \hat{j}+\frac{\partial t'}{\partial x_k'} \hat{k}\right)\left(\frac{\partial }{\partial t'}\right) \right \rbrace[\rho]_{ret}\\
=[\nabla'\rho]_{ret} + \nabla'(t')\left[\frac{\partial \rho}{\partial t'}\right]_{ret}$$
Where the $\nabla'$ acts while keeping $t'$ fixed and the derivative wrt $t'$ is taken while keeping primed spatial coordinates fixed. So,
$$[\nabla' \rho]_{ret} = \nabla'[\rho]_{ret} - \left[\frac{\partial \rho}{\partial t'}\right]_{ret} \nabla'(t-R/c)$$
Best Answer
The Fourier transform is a linear process. Therefore, it commutes with any other linear operation. What I mean by that is that we can interchange the order in which we apply these operations. Since Maxwell's equations are linear operations (provided that the medium is linear), we can apply the Fourier transform on these equations in arbitrary order.
The role of the Fourier transform is to represent the electric field for example in terms of basis functions - plane waves. These plane waves are solutions of the Helmholtz equation that follows from Maxwell's equations. So, the spectrum that we get from the Fourier transform represents the coefficient function telling us how to recover the original electric field by recombining the plane waves. So now we only need to work with the spectrum.
A vector field can be represented in terms of the individual components. When the Fourier transform is applied to a vector field, one can see it as a Fourier transform on each of the components individually. Therefore, we end up with a spectrum that is also a vector quantity.
The Poynting vector requires a product operation (the curl), which is nonlinear. When we compute the Fourier transform of the product of two functions we end up with the convolution of the spectra of the two functions. So the expression in the Fourier domain does not look the same as in space and time.