I found in some game physics engines the equation is expressed in inertia frame rather than body frame.
$$\tau=I\dot{\omega}+\omega \times I \omega$$
In that situation the inertia tensor $I$ is not constant but varying with the body frame. Therefore it is updated with the orientation of rigid body. Moment $\tau$ and angular velocity $\omega$ are expressed in world frame(inertia frame). Is the formula above established?
tl;dr: Yes, this is valid. Getting there is nontrivial.
The $\boldsymbol \omega_B \times (\mathrm I_B \, \boldsymbol \omega_B)$ term in $\boldsymbol \tau_B = \mathrm I_B \, \dot{\boldsymbol \omega}_B + \boldsymbol \omega_B \times (\mathrm I_B \, \boldsymbol \omega_B)$ results from working in a rotating frame. It is a fictitious torque, very much akin to the fictitious forces that arise with Newton's second law when computed in a rotating frame. Just as those fictitious forces vanish in an inertial frame, so does the fictitious torque. The inertial frame rotational equation of motion is
$$\boldsymbol\tau_I = \frac d{dt} (\mathrm I_I\, \boldsymbol \omega_I) =
\mathrm I_I \, \dot {\boldsymbol \omega}_I
+ \dot {\mathrm I}_I \,\boldsymbol \omega_I$$
This raises the question, what is the time derivative of the inertia tensor from the perspective of an inertial frame? To determine this, the relationship between the inertia tensor as expressed in the body frame and as expressed inertial frame is needed. This is
$$\mathrm I_I = \mathrm T_{R\to I}\,\mathrm I_B\,\mathrm T_{R\to I}^{\,T}$$
where $\mathrm I_I$ is the inertia tensor expressed in the inertial frame, $\mathrm T_{R\to I}$ is the transformation matrix from the body frame to the inertial frame, $\mathrm I_B$ the inertia tensor expressed in the body frame, and $\mathrm T_{R\to I}^{\,T}$ is the transpose of (and also the inverse of) $\mathrm T_{R\to I}$. Taking the derivative with respect to time yields
$$\dot {\mathrm I}_I =
\dot {\mathrm T}_{R\to I} \, \mathrm I_B\,\mathrm T_{R\to I}^{\,T}
+ \mathrm T_{R\to I} \, \mathrm I_B \, \dot {\mathrm T}_{R\to I}^{\,T}$$
Without derivation, the time derivative of the transformation matrix is
$$\dot {\mathrm T}_{R\to I} = \mathrm T_{R\to I} \operatorname{Sk}(\boldsymbol \omega_B) = \operatorname{Sk}(\boldsymbol \omega_I)\mathrm T_{R\to I}$$
where $\operatorname{Sk}(a)$ is the skew symmetric cross product matrix generated by the vector $\boldsymbol a$:
$$\operatorname{Sk}(\boldsymbol a) \equiv \begin{pmatrix}
\phantom{-}0 & -a_z & \phantom{-}a_y \\
\phantom{-}a_z & \phantom{-}0 & -a_x \\
-a_y & \phantom{-}a_x & \phantom{-}0
\end{pmatrix}$$
Note that $\operatorname{Sk}(\boldsymbol a) \, \boldsymbol b = \boldsymbol a \times \boldsymbol b$. With the above, the time derivative of the inertia tensor as expressed in the inertial frame is
$$\dot {\mathrm I}_I =
\operatorname{Sk}(\boldsymbol \omega_I) \mathrm T_{R\to I} \, \mathrm I_B\,\mathrm T_{R\to I}^{\,T}
+ \mathrm T_{R\to I} \, \mathrm I_B \, \mathrm T_{R\to I}^{\,T} \operatorname{Sk}^T(\boldsymbol \omega_I)
= \operatorname{Sk}(\boldsymbol \omega_I)\,\mathrm I_I - \mathrm I_I \, \operatorname{Sk}(\boldsymbol \omega_I)$$
This can be nonzero due to the non-commutative nature of matrix multiplication. We're not interested so much in the form of $\dot {\mathrm I}_I$ in general so much as the product $\dot {\mathrm I}_I\,\boldsymbol \omega_I$. This is
$$\dot {\mathrm I}_I\,\boldsymbol \omega_I =
\operatorname{Sk}(\boldsymbol \omega_I)\,\mathrm I_I \,\boldsymbol \omega_I
- \mathrm I_I \, \operatorname{Sk}(\boldsymbol \omega_I) \,\boldsymbol \omega_I
= \boldsymbol \omega_I \times (\mathrm I_I \,\boldsymbol \omega_I)
- \mathrm I_I \, (\boldsymbol \omega_I \times \boldsymbol \omega_I)
= \boldsymbol \omega_I \times (\mathrm I_I \,\boldsymbol \omega_I)$$
and thus
$$\boldsymbol\tau_I =
\mathrm I_I \, \dot {\boldsymbol \omega}_I
+ \boldsymbol \omega_I \times (\mathrm I_I \,\boldsymbol \omega_I)$$
Best Answer
No, you can't the MMOI of a "static" fluid for a rotating scheme.
If you slowly bring a rotating container of fluid up to speed you will notice the fluid will "ride-up" bringing a lot of the mass further away from the rotating axis. This will increase the MMOI dramatically.
Even though all the fluid might be moving with the same rotational velocity in a steady-state condition, the shape of the fluid will be different enough to make this sort of idealization, not a good one.
If you have a good idea of the distribution of fluid mass, then you can try to do the integral to find the MMOI of the deformed fluid using the volume of the revolution process from calculus.
For a fluid which not all parts rotate with the same rate, there is still a total angular momentum involved which you can calculate. And if you know the rotational speed of the container you can divide the two to estimate an effective MMOI for the fluid.