This is a pretty standard "mixing problem." You went wrong in a couple of places:
- Your set-up for $S'(t)$ has a wrong sign. (This get spontaneously "fixed" later on, which suggests and error in copying somewhere).
- But more seriously: Your integrating factor is incorrect.
You start pretty well: if we let $S(t)$ be the amount of salt (in grams) in the tank at time $t$ ($t$ measured in minutes). (Note, $S(t)$ is the amount, not the concentration; your formulas clearly view $S$ as the amount, not the concentration; see Pickahu's set-up if you want to use the concentration instead).
In these problems, the amount of salt at any given time is changing by the formula
$$\frac{dS}{dt} = \binom{\text{rate}}{\text{in}} - \binom{\text{rate}}{\text{out}}.$$
And the initial condition $S(0)$ depends on the problem.
The initial condition is simple enough: you are told there are 250 liters of water, with a salt concentration of 7 grams per liter. So
$$S(0) = \left(250\ \text{liters}\right)\left(7\ \frac{\text{grams}}{\text{liter}}\right) = 1750\ \text{grams of salt.}$$
What about the rates in and out? We are adding 9 liters per minute, each liter containing 3 grams of salt. That is, the rate in is:
$$\text{rate in} = \left(3\frac{\text{grams}}{\text{liter}}\right)\left(9\frac{\text{liters}}{\text{minute}}\right) = 27\frac{\text{grams}}{\text{minute}};$$
What is the rate out? We are letting out 5 liters per minute; each liter will have as much salt as the concentration at time $t$. The concentration at time $t$ is given by the amount of salt at time $t$, which is $S(t)$, divided by the amount of liquid at time $t$.
From the moment we start with $250$ liters, each minute you add $9$ liters and you drain $5$ liters, for a net total addition of $4$ liters per minute. So at time $t$, the total amount of liquid in the tank is $250+4t$. So the concentration of salt at time $t$ is
$$\frac{S(t)}{250+4t}\ \frac{\text{grams}}{\text{liter}}.$$
Since we are draining five liters at this concentration, we have that
$$\text{rate out} = \left(5\ \frac{\text{liters}}{\text{minute}}\right)\left(\frac{S(t)}{250+4t}\ \frac{\text{grams}}{\text{liter}}\right) = \frac{5S(t)}{250+4t}\ \frac{\text{grams}}{\text{minute}}.$$
So the differential equation we need to solve is:
$$\frac{dS}{dt} = 27 - \frac{5S}{250+4t}.$$
Writing this in the standard form, we have
$$S' + \frac{5}{250+4t}S = 27.$$
We need an integrating factor. Letting $\mu(t)$ stand for this factor, multiplying through we have
$$\mu(t)S' + \frac{5\mu(t)}{250+4t}S = 27\mu(t)$$
and we want to realize the left hand side as the derivative of a product; that is, we want
$$\mu'(t) = \frac{5\mu(t)}{250+4t}.$$
Separating variables we have
$$\begin{align*}
\frac{\mu'(t)}{\mu(t)} &= \frac{5}{250+4t}\\
\int\frac{d\mu}{\mu} &= \int \frac{5\,dt}{250+4t}\\
\ln|\mu| &= \frac{5}{4}\ln|250+4t| + C\\
\mu(t) &= A(250+4t)^{5/4}
\end{align*}$$
Picking $A=1$, we can take $\mu(t) = (250+4t)^{5/4}$. (Another error in your computation).
That is, we have:
$$(250+4t)^{5/4}S' + \frac{5(250+4t)^{5/4}}{250+4t}S = 27(250+4t)^{5/4}$$
or
$$(250+4t)^{5/4}S' + 5(250+4t)^{1/4}S = 27(250+4t)^{5/4}$$
which can be written as
$$\Bigl( (250+4t)^{5/4}S\Bigr)' = 27(250+4t)^{5/4}.$$
You might benefit from writing out the derivations very carefully (as I did above) rather than trying to rely on formulas (I assume that's how you tried to obtain your integrating factor $I$, which was mistakenly computed).
Can you take it from here? Careful with the integral on the right hand side.
We model the situation by a system of two homogeneous linear differential equations with constant coefficients.
Let $x=x(t)$ be the quantity, in grams, of salt in tank A at time $t$, and $y=y(t)$ the quantity of salt in tank B at time $t$.
We are given the initial conditions $x(0)=60$ and $y(0)=0$.
We now examine the flow of salt in and out of tank A. Brine is flowing out to B at the rate of $20$ liters per minute. At any time $t$, the amount of salt in a liter of water is $x/200$, so salt is flowing out at the rate $(20/200)x$. Also, salty (after a while) water is flowing in from B at rate $5$ liters per minute. Thus salt is flowing into A at the rate $(5/200)y$. The above out-in analysis for the salt can be written as the differential equation
$$\frac{dx}{dt}=-\frac{20}{200}x +\frac{5}{200}y \qquad\qquad\text{(Equation $1$)}$$
Now we do a similar analysis for tank B. Salt is flowing in from A at the rate of $(20/100)x$. Salt is flowing out to A at the rate $(5/200)y$, and to the outlet pipe at the rate $(15/200)y$, for a combined out rate of $(20/200)y$. This yields the differential equation
$$\frac{dy}{dt}=\frac{20}{200}x -\frac{20}{200}y \qquad\qquad\text{(Equation $2$)}$$
We run through the solution process. Let $M$ be the matrix
$$\begin{pmatrix} -\frac{20}{200}& \frac{5}{200}\\ \frac{20}{200}& -\frac{20}{200}
\end{pmatrix}$$
We first find the eigenvalues and associated eigenvectors of the matrix $M$. Standard calculation shows that the eigenvalues are $-1/20$ and $-3/20$. The vector $(1,2)$ (written as a column vector) is an eigenvector for eigenvalue $-1/20$; the vector $(1,-2)$, again written as a column vector, is an eigenvector for eigenvalue $-3/20$.
Theory tells us that the general solution $(x,y)$ is given by
$$(x,y)=Ce^{-t/20}(1,2)+De^{-3t/20}(1,-2)$$
where $C$ and $D$ are constants.
Less compactly,
$$x=Ce^{-t/20}+De^{-3t/20} \qquad \text{and}\qquad y=2Ce^{-t/20}-2De^{-3t/20}.$$
From the fact that $x(0)=60$ we get $C+D=60$. From $y(0)=0$ we get $2C-2D=0$. So $C=D=30$, and therefore
$$x=30e^{-t/20}+30e^{-3t/20} \qquad \text{and}\qquad y=60e^{-t/20}-60e^{-3t/20}.$$
Now we know everything, so we can answer any question. We were asked when there are equal percentages of salt in both tanks. The tanks are of equal size, so equal percentage happens when the amounts of salt are equal, that is, at the time $t$ when $x=y$. Thus
$$30e^{-t/20}+30e^{-3t/20}=60e^{-t/20}-60e^{-3t/20}.$$
This simplifies to $3e^{-3t/20}=e^{-t/20}$, then to $e^{t/10}=3$. Take the natural logarithm of both sides: $t=10\ln 3\approx 10.98$.
Best Answer
You do the same thing, only now the volume is also variable, so that for the "salt mass" = "salt concentration" times "volume" you get $$ x(t+Δt)V(t+Δt)=x(t)V(t)-x(t)V_{out}Δt \\ V(t+Δt)=V(t)+(V_{in}-V_{out})Δt $$ so that applying the product rule of differentiation $$ \dot x(t)V(t)+x(t)\dot V(t)=-x(t)V_{out} \\ \dot V(t)=V_{in}-V_{out} $$ where $V_{out}=20L/min$ and $V_{in}-V_{out}=-10L/min$. This gives directly $V(t)=400-10t$, $t$ in minutes.
So it remains to solve a nice linear ODE $$ (400-10t)\dot x(t)+10x(t)=0 \implies x(t)=C(400-10t). $$ By the initial conditions $C=0.005/400=0.01/800$. As density and volume shrink linearly, their product, the total mass of salt in the tank, follow the quadratic law $$ x(t)V(t)=\frac{(40-t)^2}{800}. $$