If there are no boundedness conditions on the complexes then this need not be true. For example, let $F_\bullet=A_\bullet$ be the unbounded complex
$$\cdots\xrightarrow{\times 2}\mathbb{Z}/4\mathbb{Z}\xrightarrow{\times 2}\mathbb{Z}/4\mathbb{Z}\xrightarrow{\times 2}\mathbb{Z}/4\mathbb{Z}\xrightarrow{\times 2}\cdots$$
of $\mathbb{Z}/4\mathbb{Z}$-modules (so it is both exact and a complex of flat $\mathbb{Z}/4\mathbb{Z}$-modules).
Then each component of $F_\bullet\otimes_{\mathbb{Z}/4\mathbb{Z}}A_\bullet$ is isomorphic to
$$\bigoplus_{i\in\mathbb{Z}}\mathbb{Z}/4\mathbb{Z},$$
the kernel of the differential is
$$\bigoplus_{i\in\mathbb{Z}}2\mathbb{Z}/4\mathbb{Z},$$
and the image of the differential is
$$\left\{(a_i)_{i\in\mathbb{Z}}\in\bigoplus_{i\in\mathbb{Z}}2\mathbb{Z}/4\mathbb{Z}
\,\middle\vert\,\sum_{i\in\mathbb{Z}}a_i=0\right\}.$$
So $F_\bullet\otimes_{\mathbb{Z}/4\mathbb{Z}}A_\bullet$ has homology isomorphic to $2\mathbb{Z}/4\mathbb{Z}$ in each degree, and in particular is not exact.
However, with suitable boundedness conditions, it is true.
If $F_\bullet$ is bounded, then an induction argument on the length of $F_\bullet$ shows that $F_\bullet\otimes A_\bullet$ is exact. From this one can prove that $F_\bullet\otimes A_\bullet$ is exact if
(a) either $F_\bullet$ or $A_\bullet$ is bounded, or
(b) both $F_\bullet$ and $A_\bullet$ are bounded above, or
(c) both $F_\bullet$ and $A_\bullet$ are bounded below.
[Alternatively, spectral sequence arguments can be used.]
Similar results hold for the internal Hom, but conditions (b) and (c) are replaced by the condition that one of $F_\bullet$ and $A_\bullet$ is bounded below, and the other is bounded above.
The problem is that Hatcher does not properly define the concepts of chain maps and chain homotopies. He introduces them "en passant" when considering induced maps $f_\sharp :C_n(X) \to C_n(Y)$ and prism operators $P : C_n(X) \to C_{n+1}(Y)$ and leaves the technical details of a general definition to the reader.
It is clear that a chain map $\mathbf f : \mathbf C \to \mathbf D$ between chain complexes $\mathbf C, \mathbf D$ is a collection of group homomorphisms $f_n : C_n \to D_n$, $n \ge 0$, such that $\partial_{D,n} f_{n} = f_{n-1} \partial_{C,n}$ for all $n \ge 0$.
According to Hatcher on p. 113, a chain homotopy $\mathbf P : \mathbf C \to \mathbf D$ between chain maps $\mathbf f, \mathbf g : \mathbf C \to \mathbf D$ is a collection of group homomorphisms $P_n:C_n \to D_{n+1}$ such that $f_n - g_n = \partial_{n+1} P_n + P_{n-1}\partial_n$ for all $n$; but he is not specific about $n$. Clearly we need it for all $n \ge 0$ and this requires $P_{-1} : C_{-1} \to D_0$. Though not explicitly stated, it should be clear that Hatcher understands $C_{-1} = 0$ since this is the range of $\partial_0$. And since the zero map is the unique map $0 \to D_0$, we automatically have $P_{-1} = 0 : 0 \to D_0$.
Thus, formally, $\mathbf P$ is a collection of group homomorphisms $P_n:C_n \to D_{n+1}$, $n \ge -1$, such that $f_n - g_n = \partial_{n+1} P_n + P_{n-1}\partial_n$ for all $n \ge 0$.
A more general approach would be to define a chain complex as a collection of abelian groups $C_n$ and group homomorphisms $\partial_n : C_n \to C_{n-1}$, $n \in \mathbb Z$, such that $\partial_n \partial_{n+1} = 0$ for all $n$. It should now be obvious how to define chain maps and chain homotopies for such general chain complexes. Then we can say that the chain complexes occuring in simplicial and singular homology are "nonnegative" which means that $C_n = 0$ for $n < 0$.
Best Answer
As preliminary computation, I suggest you to consider the classical example of Koszul complex $(K,d):=(S(X^{*})\otimes \wedge (X^{*}), d)$, where $X$ is a finite dim. vector space over $\mathbb R$ or $\mathbb C$ and $S(X^{*})$, resp. $\wedge (X^{*})$ denote the symmetric resp. exterior algebra over $X$.
It is helpful because you can consider the symmetric and the exterior algebras as suitable algebra of polynomials and the construction is really explicit.
Let us specify that the Koszul complex is negatively graded, i.e.
$K^{-p}:= S(X^{*})\otimes \wedge^{p} (X^{*}) \rightarrow S(X^{*})\otimes \wedge^{p-1} (X^{*}) \rightarrow \dots \rightarrow S(X^{*})\otimes \wedge^{0} (X^{*}) \rightarrow \mathbb K$
in order to have a degree $+1$ differential $d$, which is given (on monomials in $\wedge^{p} (X^{*})$) by
$d(q \otimes (x_1 \wedge\dots\wedge x_p)):=\sum_{i=1}^{p}(-1)^{i-1} q\cdot x_i \otimes (x_i\wedge\dots \wedge\hat{x}_i\wedge\dots\wedge x_p))$, where $\hat{\cdot}$ denotes omission and $q\cdot x_i$ is the product in the symmetric algebra of the polynomial $q$ and the monomial $x_i$. The sign $(-1)^{i-1}$ comes from the fact that $x_i$ "surpasses" $x_1\wedge\dots \wedge x_{i-1}$ to reach the polynomial $q$ on the left.
This formula for the differential is exactly the one which appears in the Wiki-article on the Koszul complex you mention in your question. The reduction to your case is now straightforward.