This is a pretty standard "mixing problem." You went wrong in a couple of places:
- Your set-up for $S'(t)$ has a wrong sign. (This get spontaneously "fixed" later on, which suggests and error in copying somewhere).
- But more seriously: Your integrating factor is incorrect.
You start pretty well: if we let $S(t)$ be the amount of salt (in grams) in the tank at time $t$ ($t$ measured in minutes). (Note, $S(t)$ is the amount, not the concentration; your formulas clearly view $S$ as the amount, not the concentration; see Pickahu's set-up if you want to use the concentration instead).
In these problems, the amount of salt at any given time is changing by the formula
$$\frac{dS}{dt} = \binom{\text{rate}}{\text{in}} - \binom{\text{rate}}{\text{out}}.$$
And the initial condition $S(0)$ depends on the problem.
The initial condition is simple enough: you are told there are 250 liters of water, with a salt concentration of 7 grams per liter. So
$$S(0) = \left(250\ \text{liters}\right)\left(7\ \frac{\text{grams}}{\text{liter}}\right) = 1750\ \text{grams of salt.}$$
What about the rates in and out? We are adding 9 liters per minute, each liter containing 3 grams of salt. That is, the rate in is:
$$\text{rate in} = \left(3\frac{\text{grams}}{\text{liter}}\right)\left(9\frac{\text{liters}}{\text{minute}}\right) = 27\frac{\text{grams}}{\text{minute}};$$
What is the rate out? We are letting out 5 liters per minute; each liter will have as much salt as the concentration at time $t$. The concentration at time $t$ is given by the amount of salt at time $t$, which is $S(t)$, divided by the amount of liquid at time $t$.
From the moment we start with $250$ liters, each minute you add $9$ liters and you drain $5$ liters, for a net total addition of $4$ liters per minute. So at time $t$, the total amount of liquid in the tank is $250+4t$. So the concentration of salt at time $t$ is
$$\frac{S(t)}{250+4t}\ \frac{\text{grams}}{\text{liter}}.$$
Since we are draining five liters at this concentration, we have that
$$\text{rate out} = \left(5\ \frac{\text{liters}}{\text{minute}}\right)\left(\frac{S(t)}{250+4t}\ \frac{\text{grams}}{\text{liter}}\right) = \frac{5S(t)}{250+4t}\ \frac{\text{grams}}{\text{minute}}.$$
So the differential equation we need to solve is:
$$\frac{dS}{dt} = 27 - \frac{5S}{250+4t}.$$
Writing this in the standard form, we have
$$S' + \frac{5}{250+4t}S = 27.$$
We need an integrating factor. Letting $\mu(t)$ stand for this factor, multiplying through we have
$$\mu(t)S' + \frac{5\mu(t)}{250+4t}S = 27\mu(t)$$
and we want to realize the left hand side as the derivative of a product; that is, we want
$$\mu'(t) = \frac{5\mu(t)}{250+4t}.$$
Separating variables we have
$$\begin{align*}
\frac{\mu'(t)}{\mu(t)} &= \frac{5}{250+4t}\\
\int\frac{d\mu}{\mu} &= \int \frac{5\,dt}{250+4t}\\
\ln|\mu| &= \frac{5}{4}\ln|250+4t| + C\\
\mu(t) &= A(250+4t)^{5/4}
\end{align*}$$
Picking $A=1$, we can take $\mu(t) = (250+4t)^{5/4}$. (Another error in your computation).
That is, we have:
$$(250+4t)^{5/4}S' + \frac{5(250+4t)^{5/4}}{250+4t}S = 27(250+4t)^{5/4}$$
or
$$(250+4t)^{5/4}S' + 5(250+4t)^{1/4}S = 27(250+4t)^{5/4}$$
which can be written as
$$\Bigl( (250+4t)^{5/4}S\Bigr)' = 27(250+4t)^{5/4}.$$
You might benefit from writing out the derivations very carefully (as I did above) rather than trying to rely on formulas (I assume that's how you tried to obtain your integrating factor $I$, which was mistakenly computed).
Can you take it from here? Careful with the integral on the right hand side.
Let $y(t)$ be the kg's of salt in the tank at time $t$, and let $A$ be the kgs of salt initially in the tank. We need the change in $y(t)$.
I think it's best to use "(rate-in)(concentration in) - (rate-out)(concentration out)." Your rate in is $12$ l/m and the concentration is $2.5$ kg/l. So the amount of salt entering the tank is $12\cdot 2.5 = 30$. The rate out is $8$ l/m. The concentration out is trickier. It equals the amount of sale in the tank at time $t$ (call it $y(t)$) divided by the amount of water, which will be $400+4t$ because we start with $400$ liters and have a net increase of $4 (=12-8)$ liters every minute. So the concentration out is $y/(400+4t)$.
The to write the IVP, we set $y'$ equal to all the changes in $y$:
$$y' = 30 - \frac{y}{400+4t} \; y(0) =A.$$
This is a linear IVP, which is straightforward to solve, but $A$ will be involved in the solution. Plug $200$ in for $y$ and $60 (= 1$ hour) for $t$ and solve for $A$.
Best Answer
While there is solution being added to the tank at a rate of $4$ liters per minute, the tank is also being drained at a rate of $4$ liters per minute, so the volume stays constant at $100$ liters, throughout the process.