$\DeclareMathOperator{\Aut}{Aut}
\newcommand{\p}{\pi_1X}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\XX}{\tilde{X} }$Ok, I finally figured it out! So, the cellular chain complex of $\XX$ is given by
$$ 0\longrightarrow \Z[t, t^{-1}] \overset{\delta}\longrightarrow \Z[t, t^{-1}] \longrightarrow 0, $$
where the boundary map $\delta$ is just multiplication by $t-1$, for geometrical reasons.
The new boundary map $D:\Z_3\to\Z_3$ which we obtain afte tensoring with $\Z_3$ over $\Z[t,t^{-1}]$ can be computed by looking at the sequence of maps
$$
\Z_3
\overset{\cong}\longrightarrow
\Z[t,t^{-1}]\underset{\Z[t,t^{-1}]}\otimes\Z_3 \;
\overset{\delta\otimes id}\longrightarrow
\Z[t,t^{-1}]\underset{\Z[t,t^{-1}]}\otimes\Z_3
\overset{\cong}\longrightarrow
\Z_3,
$$
where the first map is $a\mapsto 1\otimes a$ and last map is $1\otimes a \mapsto a $. Of course the point is that we need reduce to the form $1\otimes a$ before applying the last map. Therefore we get
$$
a \mapsto
1\otimes a
\mapsto (t-1)\otimes a = 1 \otimes (ta - a)
\mapsto ta - a.
$$
Hence by a direct computation $D$ turns out to be the identity of $\Z_3$:
$$ D(0) = 0, \quad D(1)=t\cdot1 - 1 = 2-1 = 1, \quad D(2)=t\cdot 2 - 2 = 1-2 \equiv 2 \mod 3. $$
Therefore we conclude that the homology groups of $S^1$ with coefficients in $\Z_3$ twisted by the nontrivial $\rho$ are trivial:
$$ H_0(S^1\Z_3)_\rho \cong H_1(S^1\Z_3)_\rho \cong 0 $$
To answer your first question, one way to define (co)homology with local coefficients is the following.
Let $X$ be a space, let $\Pi(X)$ be the fundamental groupoid of $X$, ie. a category with objects points of $X$ and morphisms $x \rightarrow y$ given by homotopy classes of paths. A local coefficient system $M$ on $X$ is a functor $M: \Pi(X) \rightarrow \mathcal{A}b$ from the fundamental groupoid to the category of abelian groups. In particular, $M$ associates a "group of coefficients" $M(x)$ to every point of $x \in X$.
Associated to $M$ is the singular complex given by
$C _{n}(X, M)= \bigoplus _{\sigma \in Sing_{n}(X)} M(\sigma(1,0,\ldots,0))$,
where $Sing _{n}(X)$ is the set of all maps $\Delta ^{n} \rightarrow X$. The differential can be defined using the fact that the groups $M(x)$ are functorial with respect to paths in $X$. Observe that this is a very similar to the usual definition of singular complex with coefficients in an abelian group $A$, which would be
$C _{n}(X, A) = \bigoplus _{\sigma \in Sing_{n}(X)} A$,
except in the "non-local" case, we count the occurances of any $\sigma: \Delta^{n} \rightarrow X$ in a given chain using the same group $A$ and in the local case, we use $M(\Delta^{n}(1, 0, \ldots, ))$, which might be different for different $\sigma$.
This is the locality (localness?) in the name, which should be contrasted with globality of usual homology with coefficients, where the choice of the group $A$ is global and the same for all points.
The definition I have given above is enlightening but perhaps not suitable for computations. Luckily under rather weak assumptions one can use the definition you allude to. Let me explain. Let $X$ be path-connected, let $x \in X$ and consider $\pi = \pi_{1}(X, x)$ as a category with one object and morphisms the elements of the group.
The obvious inclusion $\pi \hookrightarrow \Pi(X)$ is an equivalence of categories and so the functor categories $[\pi, \mathcal{A}b], [\Pi(X), \mathcal{A}b]$ are equivalent, too. But the left functor category is exactly the category of $Z[\pi]$-modules! In particular, we have a bijection between isomorphism classes of local coefficient systems on $X$ and $Z[\pi]$-modules.
If $X$ is nice enough to admit a universal cover $\tilde{X}$, the above allows us to give another definiton of homology with local coefficients, the one you know. Let $M$ be a local coefficient system and let $M^\prime$ be the associated $\mathbb{Z}[\pi]$-module under the above equivalence (which is unique up to a unique isomorphism). Since $\pi$ acts on $\tilde{X}$, it also acts on $C _{\bullet}(X, \mathbb{Z})$ and so the latter is a chain complex of $\mathbb{Z}[\pi]$-modules. We the can define homology with local coefficients to be homology of the complex
$C_{n}(X, M ^\prime) = C_{n}(\tilde{X}, \mathbb{Z}) \otimes _{\mathbb{Z}[\pi]} M^\prime$
One can show that this two definitions I gave agree, that is, for $X$ like above we have an isomorphism $H_{n}(X, M) \simeq H_{n}(X, M^\prime)$.
Best Answer
You are almost correct, the problem is in your assumption that a fine enough triangulation will yield that $G$ respects the boundary morphisms.
In fact we don't need to work cellularly, this works just as well with the singular chain complex (because $\Delta^n$ is contractible, so up to choosing a basepoint we have unique liftings $\Delta^n\to X$ to $\Delta^n\to X'$), and I'll rather use this to show how the differentials differ (it seems clearer to me that way)
Also, for simplicity, I'll take $\pi' = 0$ so $X' = \tilde X$.
Now we need to analyze the map $C_n(\tilde X)\otimes_{\mathbb Z[G]}\otimes M \to C_n(X;M)$. For each $\sigma : \Delta^n\to X$, fix one $\tilde \sigma : \Delta^n\to \tilde X$ that lifts it. By the usual lifting property, this is equivalent to choosing a lift $p_{\sigma}$ of $\sigma(e_0)$ for each $\sigma$, and this will come in handy.
This choice is what gives you the isomorphism $C_n(\tilde X) = \bigoplus_{\sigma : \Delta^n\to X}\mathbb Z[G]$
Now your map takes $\tau\otimes m$, writes it as $(g\cdot \tilde\sigma) \otimes m$ for some $g\in \pi, \sigma : \Delta^n \to X$, and then sends this to $\sigma \otimes g\cdot m$ (where I see $C_n(X;M) = C_n(X;\mathbb Z)\otimes M$). Note that $\sigma = p\circ \tau$ so it's almost $p\circ \sigma \otimes m$, but it's twister by the difference
Now look at $\partial (\tau\otimes m) = \partial \tau \otimes m = \sum_{i=0}^n (-1)^i d_i\tau \otimes m$, write $d_i\tau$ as $g_i\cdot (\widetilde{p\circ d_i\tau})$, and this is sent to $\sum_{i=0}^n (-1)^i p\circ d_i\tau \otimes g_i\cdot m$
Compare to what happens if you first send it over to $p\circ \tau\otimes g\cdot m$ ($g$ such that $g\cdot \widetilde{p\circ\tau} = \tau$), and then take the boundary : although clearly $d_i(p\circ \tau) = p\circ d_i\tau$, you have $g\cdot m$ instead of the varying $g_i$ : you get $\sum_{i=0}^n (-1)^i p\circ d_i\tau \otimes g\cdot m$
So a priori there's no reason for the two differentials to agree, they're somehow "twisted" by $\pi$, and that's precisely what local coefficients do : they twist the differential.
In fact, what you can do is, instead of fixing $p_\sigma$ for each $\sigma$, fix an antecedent $p_x$ for each $x\in X$. Then we define $\tilde\sigma$ to be the unique lift of $\sigma$ that sends $e_0$ to $p_{\sigma(e_0)}$ : this gives something more uniform, and you can check that in this situation, the differentials are almost the same : for $i\neq 0$, $g_i = g$, but for $i=0$, there's a problem in that $d_0\tau$ doesn't have the same evaluation in $e_0$ as $\tau$.
More precisely, let $\tau = \tilde\sigma$ (so that there's no need for a $g$). Then for each $i$, $d_i\tau$ is a lift of $d_i\sigma$, and for $i\neq 0$, $d_i\tau (e_0) = \tau(e_0) = p_{\sigma(e_0)}$, so that $d_i\tau = \widetilde{d_i\sigma}$ : there is no $g_i$. But for $i=0$, $d_0\tau(e_0) = \tau(e_1)$ which is not necessarily $p_{\sigma(e_1)}$ : it is related to it by some $g\in \pi$ (because it's also a lift of $\sigma(e_1)$), so that one of the differentials looks like $p\circ d_0\tau \otimes g\cdot m+ \sum_{i=1}^n (-1)^i p\circ d_i\tau \otimes m$, and the other is $\sum_{i=0}^n (-1)^i p\circ d_i\tau \otimes m$.
So they're almost the same except for a twist on the $0$th face - this is the same twist as the one in something called the standard resolution, when computing group (co)homology, if you know about that.
Now you can clearly (hopefully) see that there's no reasonable condition to impose on the triangulation or the cell-structure or whatever that will make sure that this twist disappears and that the two differentials actually agree.