In the Lagrangian formalism of path integral quantization, one invokes the formula $$\int d\theta\,\delta(f(\theta))=\int df\det\Bigg|\frac{d\theta}{df}\Bigg|\delta(f(\theta))=\det\Bigg|\frac{d\theta}{df}\Bigg|_{f=0}. \tag{$\ast$}$$
Now consider the naive partition function $$\mathcal{Z}=\int\mathcal{D}[A]e^{-S[A]}. \tag{1}$$
The action $S[A]$ and the functional integral measure $\mathcal{D}[A]$ must be gauge invariant under an arbitrary gauge transformation $$A[U]=U^{-1}dU+U^{-1}AU\quad\mathrm{or}\quad\delta_{U}A=d\,\delta U+[A,\delta U], \tag{$\star$}$$
and $\mathcal{D}[A]\equiv\prod_{\mu,a,x}dA^{a}_{\mu}(x)$.
Next, consider a "haar measure" on the (infinite dimensional) Lie group of gauge transformations ($\star$), i.e $$\mathcal{D}[U]=\mathcal{D}[UU^{\prime}];\quad\mathrm{and}\quad\mathcal{D}[U]\equiv\prod_{x}dU(x),$$
and define the Faddeev-Popov determinant via the following functional integral identity $$1=\Delta[A]\int\mathcal{D}[U]\,\delta[\mathcal{F}(A[U])], \tag{2}$$
where $\mathcal{F}(A[U])$ is known as the gauge fixing condition to be specified.
Claim: The Faddeev-Popov determinant $\Delta[A]$ is gauge invariant.
proof: It can be shown as follows $$\Delta[A]^{-1}=\int\mathcal{D}[U]\,\delta[\mathcal{F}(A[U])],\quad\mathrm{hence}\quad\Delta[A[U]]^{-1}=\int\mathcal{D}[U^{\prime}]\,\delta[\mathcal{F}(A[UU^{\prime}])].$$
But since $\mathcal{D}U$ is gauge invariant, one has $$\Delta[A[U]]^{-1}=\int\mathcal{D}[UU^{\prime}]\,\delta[\mathcal{F}(A[UU^{\prime}])]=\Delta[A]^{-1}.$$
Now insert the functional integral identity (2) into the partition function (1), one has
\begin{align}
\mathcal{Z}&=\int\mathcal{D}[A]e^{-S[A]} \\
&=\int\mathcal{D}[A]\Delta[A]\int\mathcal{D}[U]\,\delta[\mathcal{F}(A[U])]e^{-S[A]} \\
&=\int\mathcal{D}[U]\int\mathcal{D}[A]\left(\Delta[A]\delta[\mathcal{F}(A[U])]e^{-S[A]}\right).
\end{align}
Now observe that the integrand $$\int\mathcal{D}[A]\left(\Delta[A]\delta[\mathcal{F}(A[U])]e^{-S[A]}\right)$$
under the integral $\int\mathcal{D}[U]$ is actually independent of $U$, therefore it can be replaced by $$\int\mathcal{D}[A[U]]\left(\Delta[A[U]]\delta[\mathcal{F}(A[U])]e^{-S[A[U]]}\right)=\int\mathcal{D}[A]\left(\Delta[A]\delta[\mathcal{F}(A)]e^{-S[A]}\right).$$
Thus, the partition function can be written as $$\mathcal{Z}=\int\mathcal{D}[U]\int\mathcal{D}[A]\left(\Delta[A]\delta[\mathcal{F}(A)]e^{-S[A]}\right). \tag{3}$$
Next, use the functional version of the formula ($\ast$), one has
\begin{align}
\Delta[A]^{-1}&=\int\mathcal{D}[\mathcal{F}]\mathrm{Det}\Bigg|\frac{\delta U}{\delta\mathcal{F}(A[U])}\Bigg|\delta[\mathcal{F}] \\
&=\int\mathcal{D}[\mathcal{F}]\mathrm{Det}\Bigg|\frac{\delta\mathcal{F}(A[U])}{\delta U}\Bigg|^{-1}\delta[\mathcal{F}] \\
&=\mathrm{Det}\Bigg|\frac{\delta\mathcal{F}(A[U])}{\delta U}\Bigg|^{-1}\Bigg|_{\mathcal{F}(A[U])=0},
\end{align}
i.e $$\Delta[A]=\mathrm{Det}\Bigg|\frac{\delta\mathcal{F}(A[U])}{\delta U}\Bigg|_{\mathcal{F}(A[U])=0}.$$
One can pick a gauge fixing condition $\mathcal{F}(A[U])$ such that $\mathcal{F}(A[U])=0$ at $U=\mathrm{id}$.
Since from the above expression one finds that only infinitesimal gauge transformations are relevant in calculation, one can safely assume $$U(x)=\exp\left\{i\sum_{a}T^{a}\Lambda_{a}(x)\right\}, \tag{4}$$
where $\left\{T_{a}\right\}_{a=1,\cdots,N}$ is a basis of the Lie algebra of the gauge group. Then, the Fadeev-Popov determinant can be chosen such that $$\Delta[A]=\mathrm{Det}\Bigg|\frac{\delta\mathcal{F}(A[U])}{\delta U}\Bigg|_{U=\mathrm{id}}$$
Then, using the functional chain-rule and linearity of determinant, one has
$$\frac{\delta\mathcal{F}(A[U](x))}{\delta U(y)}\Bigg|_{U=\mathrm{id}}=\sum_{a}\int d^{4}z\frac{\delta\mathcal{F}(A[U](x))}{\delta\Lambda_{a}(z)}\Bigg|_{\Lambda=0}\frac{\delta\Lambda_{a}(z)}{\delta U(y)}\Bigg|_{\Lambda=0}.$$
Notice that the factor on the right is the inverse of $$\left(\frac{\delta U(y)}{\delta\Lambda_{a}(z)}\right)\Bigg|_{\Lambda=0}=iT^{a}\delta(y-z),$$
which is a Lie algebra-valued constant. The above equation is an (infinite-dimensional) linear transformation on $$\mathcal{M}\equiv\frac{\delta\mathcal{F}(A[U])}{\delta\Lambda}.$$
Thus, up to some infinite constant, one can replace $\Delta[A]$ by $\mathrm{Det}\mathcal{M}$ in the partition function.
More precisely, one has $$\Delta[A]=\mathrm{Det}\Bigg|\frac{\delta\mathcal{F}(A[U])}{\delta U}\Bigg|_{U=\mathrm{id}}=\mathrm{Det}(\mathcal{M}\cdot𝟙^{-1})=\mathrm{Det}\mathcal{M},$$
where
\begin{align}
(\mathcal{M}\cdot𝟙^{-1})(x,y)&=\int d^{4}z\frac{\delta\mathcal{F}(A[U](x))}{\delta\Lambda(z)}\Bigg|_{\Lambda=0}\cdot\frac{\delta\Lambda(z)}{\delta U(y)}\Bigg|_{\Lambda=0} \\
&=\int d^{4}z\mathcal{M}(x,z)𝟙^{-1}(z,y).
\end{align}
Using the functional chain-rule, one obtains
\begin{align}
\mathcal{M}^{ab}(x,y)&\equiv\int d^{4}z\left(\frac{\delta\mathcal{F}^{a}(A[U](x))}{\delta A[U]^{c}_{\mu}(z)}\frac{\delta A[U]^{c}_{\mu}(z)}{\delta\Lambda_{b}(y)}\right)\Bigg|_{\Lambda=0} \\
&=\int d^{4}z\frac{\delta\mathcal{F}^{a}(A(x))}{\delta A^{c}_{\mu}(z)}\left(\frac{\partial}{\partial z_{\mu}}\delta^{cb}+\sum_{d}f^{cbd}A_{d\mu}(z)\right)\delta(z-y). \tag{5}
\end{align}
Also notice that under the change of variables, $$\mathcal{D}[U]=\mathrm{Det}\left(\frac{\delta U}{\delta\Lambda}\right)\mathcal{D}[\Lambda],$$
the Jacobian factor is an infinite constant (independent of gauge fields $A_{\mu}^{a}(x)$), which can be omitted in the functional integral.
Plugging (5) back into (3), one has $$\mathcal{Z}=\int\mathcal{D}[\Lambda]\int\mathcal{D}[A]\left(\mathrm{Det}\mathcal{M}\cdot\delta[\mathcal{F}(A)]\cdot e^{-S[A]}\right).$$
But the integrand $$\int\mathcal{D}[A]\left(\mathrm{Det}\mathcal{M}\cdot\delta[\mathcal{F}(A)]\cdot e^{-S[A]}\right)$$
under the integral $\int\mathcal{D}[\Lambda]$ is independent of $\Lambda$, thus the integral $\int\mathcal{D}[\Lambda]$ of gauge orbits only produces an infinite constant factor, which can be omitted.
Finally, one obtains the gauge-fixed partition function $$\mathcal{Z}=\int\mathcal{D}[A]\left(\mathrm{Det}\mathcal{M}\cdot\delta[\mathcal{F}(A)]\cdot e^{-S[A]}\right).$$
In your case, one can pick up the Lorenz gauge $$\mathcal{F}(A)=\partial_{\mu}A^{\mu}=0$$
in QED, and the Faddeev-Popov matrix is $$\mathcal{M}(x,y)=\Box\,\delta(x-y)$$
which is a constant.
More generally, in the non-Abelian case, since the $A^{a}_{\mu}(z)$ appears in the Faddeev-Popov determinant, the determinant cannot be factored out in the functional integral, and produces non-trivial interactions between photons and ghost fermions.
The gauge fixing procedure is even clearer in the canonical approach of the path-integral. TO BE CONTINUED
Best Answer
Qmechanic is right, but his answer doesn't explain why we can't just consider the ghosts as physical and be done with it.
There are two main reasons why ghosts can't be considered physical.
The problem can be traced back to the kinematics of gauge-fixing. Remember how in the $U(1)$ case we had a gauge-fixing constraint (e.g. Lorenz gauge condition), which after implementing it as a quantum operator constraint $C$ selected a unique subspace $\text{ker} C$ of physical states? Well, here we deal with a similar situation, plagued by additional technical difficulties because the group is non-abelian.
The Fock space of the gauge+ghost system of the Lagrangian mentioned in your question isn't physical. It contains negative-norm states (just like in the $U(1)$ case). As an example of the negative-norm state, consider a timelike polarized gauge boson $$ \left(a_0^{\dagger}\right)^\alpha \left| 0 \right>.$$
Just like in the $U(1)$ case this can be solved by implementing a gauge condition constraint as a quantum operator and solving. However, we run into the following complication:
This can be traced back to the following fact: the current-conservation law contains a covariant derivative instead of the ordinary one, while the Lorenz gauge condition still operates with an ordinary partial derivative.
This difficulty can be successfully solved with help of BRST quantization technique. Existence of ghosts is essential for the BRST to work.
In conclusion: the S-matrix given by the quantization of the Lagrangian from your question gives the correct quantum dynamics of the quantum gauge field, but only when projected to a subspace of the naive Fock space given by the BRST cohomology. It also has a nice property of not mixing physical and unphysical degrees of freedom, meaning that we can use its full form in practical computations and only project down to the physical subspace afterwards.
That we can't use the extended Fock space is already obvious because of the two reasons given in the beginning of my answer.