Suppose the concentration at time $t$ is $c(t)$ so $c(0)=10 \; kg/m^3$.
You have $$c'(t) = - \frac{10 \;m^3/min}{1000 \; m^3} c(t)$$
so $c(t) = k \exp(-t/100)$ for some constant $k$ and from the starting condition $k = 10 \; kg/m^3$ so
$$c(t) = 10 \, \exp\left(\frac{-t}{100}\right) \; kg/m^3.$$
This is a pretty standard "mixing problem." You went wrong in a couple of places:
- Your set-up for $S'(t)$ has a wrong sign. (This get spontaneously "fixed" later on, which suggests and error in copying somewhere).
- But more seriously: Your integrating factor is incorrect.
You start pretty well: if we let $S(t)$ be the amount of salt (in grams) in the tank at time $t$ ($t$ measured in minutes). (Note, $S(t)$ is the amount, not the concentration; your formulas clearly view $S$ as the amount, not the concentration; see Pickahu's set-up if you want to use the concentration instead).
In these problems, the amount of salt at any given time is changing by the formula
$$\frac{dS}{dt} = \binom{\text{rate}}{\text{in}} - \binom{\text{rate}}{\text{out}}.$$
And the initial condition $S(0)$ depends on the problem.
The initial condition is simple enough: you are told there are 250 liters of water, with a salt concentration of 7 grams per liter. So
$$S(0) = \left(250\ \text{liters}\right)\left(7\ \frac{\text{grams}}{\text{liter}}\right) = 1750\ \text{grams of salt.}$$
What about the rates in and out? We are adding 9 liters per minute, each liter containing 3 grams of salt. That is, the rate in is:
$$\text{rate in} = \left(3\frac{\text{grams}}{\text{liter}}\right)\left(9\frac{\text{liters}}{\text{minute}}\right) = 27\frac{\text{grams}}{\text{minute}};$$
What is the rate out? We are letting out 5 liters per minute; each liter will have as much salt as the concentration at time $t$. The concentration at time $t$ is given by the amount of salt at time $t$, which is $S(t)$, divided by the amount of liquid at time $t$.
From the moment we start with $250$ liters, each minute you add $9$ liters and you drain $5$ liters, for a net total addition of $4$ liters per minute. So at time $t$, the total amount of liquid in the tank is $250+4t$. So the concentration of salt at time $t$ is
$$\frac{S(t)}{250+4t}\ \frac{\text{grams}}{\text{liter}}.$$
Since we are draining five liters at this concentration, we have that
$$\text{rate out} = \left(5\ \frac{\text{liters}}{\text{minute}}\right)\left(\frac{S(t)}{250+4t}\ \frac{\text{grams}}{\text{liter}}\right) = \frac{5S(t)}{250+4t}\ \frac{\text{grams}}{\text{minute}}.$$
So the differential equation we need to solve is:
$$\frac{dS}{dt} = 27 - \frac{5S}{250+4t}.$$
Writing this in the standard form, we have
$$S' + \frac{5}{250+4t}S = 27.$$
We need an integrating factor. Letting $\mu(t)$ stand for this factor, multiplying through we have
$$\mu(t)S' + \frac{5\mu(t)}{250+4t}S = 27\mu(t)$$
and we want to realize the left hand side as the derivative of a product; that is, we want
$$\mu'(t) = \frac{5\mu(t)}{250+4t}.$$
Separating variables we have
$$\begin{align*}
\frac{\mu'(t)}{\mu(t)} &= \frac{5}{250+4t}\\
\int\frac{d\mu}{\mu} &= \int \frac{5\,dt}{250+4t}\\
\ln|\mu| &= \frac{5}{4}\ln|250+4t| + C\\
\mu(t) &= A(250+4t)^{5/4}
\end{align*}$$
Picking $A=1$, we can take $\mu(t) = (250+4t)^{5/4}$. (Another error in your computation).
That is, we have:
$$(250+4t)^{5/4}S' + \frac{5(250+4t)^{5/4}}{250+4t}S = 27(250+4t)^{5/4}$$
or
$$(250+4t)^{5/4}S' + 5(250+4t)^{1/4}S = 27(250+4t)^{5/4}$$
which can be written as
$$\Bigl( (250+4t)^{5/4}S\Bigr)' = 27(250+4t)^{5/4}.$$
You might benefit from writing out the derivations very carefully (as I did above) rather than trying to rely on formulas (I assume that's how you tried to obtain your integrating factor $I$, which was mistakenly computed).
Can you take it from here? Careful with the integral on the right hand side.
Best Answer
You can add a fourth line to your matrix, representing "tank 4" which is really the outside world. Why are the values in your matrix so small instead of being the single digit whole numbers of the problem?
Added: OK, I see that. So presumably you start with pure water in the tanks and are trying to calculate the salt concentration as a function of time. Now the matrix multiply is making sense-let $x_1,x_2,x_3$ be kg of salt in each tank. One equation would be $x_1'=1.5-0.07x_1+0.02x_2+0.02x_3$ so putting the outflows on the diagonal is correct. You should just add a column vector which is the input from the outside world, getting $$\begin {bmatrix} x_1'\\x_2'\\x_3'\end {bmatrix}= \begin {bmatrix}-.07&.02&.02\\ .04&-.04&0\\ .03&.02&-.05\end {bmatrix}\begin {bmatrix}x1\\ x2\\ x3\end {bmatrix}+\begin {bmatrix}1.5\\ 0\\ 0\end {bmatrix}$$