If $\mathcal H$ is the Hilbert space of the QFT, then
\begin{align}
U:\mathrm{SO}(3,1)\to \mathscr U(\mathcal H)
\end{align}
where $\mathscr U(\mathcal H)$ is the set of unitary operators on $\mathcal H$. In other words, $U$ is a unitary representation on the Hilbert space of the theory. If $V$ is the target space of the fields, then
\begin{align}
R:\mathrm{SO}(3,1)\to \mathrm{GL}(V),
\end{align}
where $\mathrm{GL}(V)$ is the vector space of invertible linear operators on $V$. In other words, $R$ is a representation on the target space of the fields in the theory. The mapping
\begin{align}
x\to \Lambda x
\end{align}
is the defining representation of $\mathrm{SO}(3,1)$ on $\mathbb R^{3,1}$. The statement that the field transforms in a particular way, namely the equation you wrote down, simply says that the action by conjugation of the Lorentz group on field operators by $U$ agrees with the composite of the representation on the target space and the inverse of the defining representaton.

The electromagnetic potential $A^\mu$ is a four-vector, and hence transforms in the fundamental representation of $\mathrm{SO}(1,3)$, i.e. $A^\mu\mapsto \Lambda^\mu_\nu A^\nu$ where $\Lambda$ is the usual 4x4 matrix associated to a Lorentz transformation.

Your question seems fundamentally confused about the difference between the *field* and the *particle*. The *field* transforms as an ordinary four-vector. A *particle* does not. This is because you have to differentiate between two different kinds of group representations here:

Each (classical) field $\phi: \mathbb{R}^4\to V$ has some target space $V$, where $V$ is a finite-dimensional representation of the relevant symmetry groups. In the case of the electromagnetic potential, $V$ is the fundamental representation - the "vector representation" - of the Lorentz group $\mathrm{SO}(1,3)$.

Upon the Hilbert space of states of the quantum theory, there is another, *unitary* representation. The unitary representations of the Poincare group are necessarily infinite-dimensional (the Lorentz group has no finite-dimensional unitary representations), and the possible infinite-dimensional representations correspond to *particles* by Wigner's classification.

The finite-dimensional representation $\rho_\text{fin}$ and the unitary representation $\rho_\text{U}$ are related by one of the Wightman axioms:
$$ \rho_\text{fin}(\Lambda)\phi = \rho_\text{U}(\Lambda)\phi\rho_\text{U}(\Lambda)^\dagger\tag{1}$$
where on the l.h.s. the $\phi$ is acted on as the classical vector it was, and on the r.h.s. it is acted on as the operator-valued field upon an infinite-dimensional Hilbert space it is in the quantum theory. This relation essentially is there to ensure that the transformation of the field as an operator upon the space of states of the quantum theory is consistent with its classical transformation behaviour.

The masslessness of the photon does *not* reflect in any change of the commutation relations of the Lorentz algebra (and I don't know why you would think it would). What the masslessness does is *change the little group* (see surface of transitivity or e.g. these notes), which is the group under which the four-momentum of a particle is invariant. For massive particles, the little group is $\mathrm{SU}(2)$, but for massless ones, it is the two-dimensional Euclidean group $\mathrm{E}(2)$ (or $\mathrm{ISO}(2)$).

Different little groups classify different infinte-dimensional unitary representations, and indeed, the rather unusual little group $\mathrm{E}(2)$ of the photon leads to the standard generators of $\mathfrak{so}(1,3)$ acting rather unusually upon a photon state. In particular, since $\mathrm{SU}(2)$ is not the little group, you cannot expect that rotations act in the same manner as for massive particles.

If you are interested in a detailed derivation of the structure of the little group and its allowed projective representations, take a look at e.g. the second half of chapter 2 of *The quantum theory of fields, Vol. I* by Weinberg.

There is a subtle issue with the relation $(1)$ for massless fields. As can be shown (cf. again Weinberg), a vector field constructed out of creation/annihilation operators for a massless particle can only obey a *modified version*
$$ \rho_\text{U}(\Lambda)\phi\rho_\text{U}(\Lambda)^\dagger = \rho_\text{fin}(\Lambda)\phi + \mathrm{d}\Omega\tag{2}$$
for a function of spacetime $\Omega$. Therefore, to make the relation $(1)$ hold on the quantum space of states, we must demand that $A\mapsto A + \mathrm{d}\Omega$ is a *gauge symmetry* of the theory, which must be quotiented out of the naive space of state upon which $(2)$ holds to obtain the actual space of physical states. Taking the quotient by identifying gauge-related states (and operators), $(2)$ becomes $(1)$ on the actual space of states (since $A+\mathrm{d}\Omega = A$ after taking the quotient). This shows that every massless vector field must be a gauge field in quantum field theory.

## Best Answer

Precisely because of the reason you have put forward,

the Lorentz transformation for a scalar field $\phi(x)$ is usually written as:

$$ \phi(x) \rightarrow \phi(x)' = \phi(\Lambda^{-1}x), $$

and for a vector field $V^{\mu}(x)$ is usually written as:

$$V^\mu(x) \rightarrow V^\mu(x)' = \Lambda^{\mu}_{\nu}\,V^{\nu}(\Lambda^{-1}x).$$

So that you leave your coordinates $x$ fixed.

Hence, your expression is correct, but it’s in terms of the rotated coordinate system $x’$. You can just write $x = \Lambda^{-1} \, x’$ in the RHS and get to my expression.

EDITFor clarity, the last expression may also be written as:

$$V^\mu(x) \rightarrow V^\mu(x')' = \Lambda^{\mu}_{\nu}\,V^{\nu}(x),$$ just by replacing $x \rightarrow x'$ and using $x = \Lambda^{-1} \, x’$.