It seems to me that you are confusing a generic notion of total derivative and the so called Lagrangian derivative (also known as material derivative).
Let us start from scratch. In Cartesian coordinates, a fluid or a generic continuous body is first of all described by a class of differentiable (smooth) maps from $\mathbb R^3$ to $\mathbb R^3$:
$$x=x(t,y)\:, \quad x \in \mathbb R^3\:.$$
For every $t\in \mathbb R$, the map $\mathbb R^3 \ni y \mapsto x(t,y) \in \mathbb R^3$ associates the particle with initial position $y$ with the position $x$ of the same particle at time $t$.
For every fixed $t$, the map above is supposed to be inverible with differentiable (smooth) inverse, and the two maps $\mathbb R^3 \ni y \mapsto x(t,y) \in \mathbb R^3$ and $\mathbb R^3 \ni x \mapsto y(t,x) \in \mathbb R^3$ are jointly smooth in all variables simultaneously.
At time $t$, the velocity of the particle with initial position $y$ is consequently given by:
$$v_L(t,y) = \frac{\partial x}{\partial t}\:.$$
The formula above pictures the field of velocities in the so called Lagrangian description: physically relevant quantities, at a given value of time, are viewed as functions of the initial position of the particles forming the continuous body.
It is however often more convenient to describe this velocity as a function of the position in space at time $t$. We obtain this way the so called Eulerian description of the field of velocities:
$$v(t,x) := v_L(t,y(t,x))\:.$$
(I will not make use on an index $E$, henceforth assuming that what it is not Lagrangian is automatically Eulerian.) $x$ is a given point in space at time $t$. At that time it is crossed by the particle with initial position $y(t,x)$.
Generally speaking, if $S$ is a (Cartesian) tensor field defined on the continuous body, we have two representations. The Lagrangian one $S_L(t,y)$ and the Eulerian one $S(t,x)$, where, obviously,
$$S(t,x) = S_L(t,y(t,x))\quad \mbox{and}\quad S_L(t,y)= S(t,x(t,y))\:.$$
It is sometime convenient to compute the derivative of $S$ along the stories of the particles of continuous body while representing the obtained tensor field in Eulerian picture.
For instance, the acceleration field is
$$a(t,x)= \left.\frac{\partial }{\partial t} v_L(t,y)\right|_{y=y(t,x)}\:.$$
Notice that the time derivative is correctly computed in Lagrangian representation, since we have to follow each particle separately.
The RHS of the identity above reads, only exploiting Eulerian objects:
$$a(t,x) = \frac{\partial v}{\partial t} + v \cdot \nabla_x v(t,x)\:.$$
(It is an easy exercise to establish that identity from definitions.)
In general the Lagrangian derivative of a tensor field (represented in Eulerian picture) $S(t,x)$ is defined as:
$$\frac{DS}{Dt} := \frac{\partial S}{\partial t} + v(t,x)\cdot \nabla_x S(t,x)\qquad(1)$$
It turs out that:
$$\left.\frac{DS}{Dt}(t,x)\right|_{x=x(t,y)}= \frac{\partial}{\partial t} S_L(t,y)\:,\qquad (2)$$
so that, the Lagrangian derivative is just a way to compute time derivatives along the stories of the particles of the continuous body remaining in Eulerian representation.
Remark. This Lagrangian derivative is the one you call total derivative, but its physical meaning is very precise as I illustrated above. In the rest of this post I keep denoting it by $D/Dt$ instead of $d/dt$.
Let us come to the conservation law of the mass. First of all, we have to endow our continuous system with a density of mass $\rho$. As before, we can adopt either an Eulerian or Lagrangian description:
$$\rho= \rho(t,x)\qquad \mbox{and}\quad \rho_L= \rho_L(t,y):= \rho(t,y(t,x))\:.$$
The most clear statement concerning conservation of mass is the following.
For every (sufficiently regular) portion $V_L$ of the initial configuration of continuous body, corresponding to a portion $V_t$ at each value $t$ of time,
$$V_t = \{x \in \mathbb R^3 \:|\: x = x(t,y)\:, y \in V_L\}\:,$$
the mass included in $V_t$ does not change in time:
$$\frac{d}{dt} \int_{V_t} \rho(t,x) dx =0\:,\qquad(3)$$
Let us transform this requirement into the equivalent local statement. Passing to the Lagrangian picture, (1) reads:
$$\frac{d}{dt} \int_{V_L} \rho(t,x(t,y)) |J_t| dy =0$$
that is
$$\frac{d}{dt} \int_{V_L} \rho_L(t,y) |J_t| dy =0$$
where $J_t$ is the determinant of the Jacobian matrix of elements $\frac{\partial x_i}{\partial y_j}$. Since in (2) $V_L$ does not depend on time and everything is regular (the integrand is smooth and $V_L$ can always be assumed to be bounded) we can swap the symbol of derivative and that of integral (essentially taking advantage of Lebesgue's dominated convergence theorem):
$$ \int_{V_L} \left(\frac{\partial}{\partial t}\rho_L(t,y)\right) |J_t| +
\rho_L(t,y) \frac{\partial}{\partial t} |J_t|
dy =0\:.$$
It is possible to prove (it is not so simple actually), that $\frac{\partial}{\partial t} |J_t| = |J_t|\nabla_x \cdot v(t,x)|_{x=x(t,y)}$. Using it in the LHS of the found identity and taking (1) and (2), into account, we find that (3) implies (in fact is equivalent to):
$$ \int_{V_L} \left( \left.\frac{D\rho}{Dt}\right|_{x=x(t,y)}+\rho_L(t,y) \left.\nabla_x \cdot v(t,x)\right|_{x=x(t,y)}\right) |J_t| dy =0\:.\qquad (4)$$
In other words, coming back to Eulerian variables:
$$ \int_{V_t} \frac{D\rho}{Dt}+\rho(t,x) \nabla_x \cdot v(t,x) dx =0\:.\qquad(5)$$
Form (4) or (5), using the fact that the integrand is continuous and $V_L$ or $V_t$ substantially are arbitrary, ones infers that the integral version of the law of mass conservation (3) is equivalent to the local requirement in Eulerian formulation:
$$\frac{D\rho}{Dt}+\rho(t,x) \nabla_x \cdot v(t,x) = 0 \qquad (6)\:.$$
Finally, makig use of the definition of Lagrangian derivative (1), that identity can equivalently be stated as:
$$\frac{\partial\rho}{\partial t}+ \nabla_x \cdot \rho(t,x) v(t,x) = 0 \qquad (7)\:.$$
Remark. A continuous body is incompressible if the volume measure $V_t$ of its portions $V_L$ remains constant along its story:
$$\frac{d}{dt} \int_{V_t} dx =0\:.\qquad(3)'$$
Dealing with as before, one immediately sees that it is completely equivalent to say that (it would be enough to everywhere replace $\rho$ for $1$)
$$\nabla_x \cdot v(t,x) = 0 \qquad (6)'\:.$$
Consequently, the law of conservation of mass for an incompressible continuous body, from (6),symply reads:
$$\frac{D\rho}{Dt}=0\:. \qquad (8)$$
This is the equation you pointed out in the case of the law of conservation of mass. However, with that interpretation, it holds for incompressible bodies only, the general formulation being (6)'.
For a generic quantity $\rho$ (even vectorial or tensorial), (8) simply says that the quantity is constant in time along the story of every particle of the system, though that constant may depend on the particle.
For the sake of completeness, let me say a few words about another popular formulation of the conservation mass law. Starting from (7), integrating both sides in a geometric volume $U$ (so, not a portion of continuous body, but a geometric volume at rest with the reference frame we are using), we have:
$$\int_U \frac{\partial \rho}{\partial t} dx = - \int_U \nabla_x \cdot \rho v(t,x) dx\:.$$
Thus, divergence theorem leads to the most popular version of the considered law whose meaning is illustrated in other answers to your question, so I will not spend any word on it:
$$\frac{d}{dt}\int_U \rho(t,x) dx = - \int_{+\partial U} \rho v(t,x)\cdot n dS(x)\:,\qquad (9)$$
where $n$ is the outward unit vector at $x\in \partial U$.
As a final remark regarding the theory of continuous bodies, let me stress that the notion of Lagrangian derivative plays a crucial role in developing the theory of continuous bodies. For instance "$F=ma$" has to be written, for every portion $V_L$ of continuous body, exploiting the Lagrangian derivative:
$$\frac{d}{dt}\int_{V_t} \rho(t,x) v(t,x) dx = F_{V_t}$$
that is, with some elementary manipulations, taking (6) into account,
$$\int_{V_t} \rho(t,x) \frac{Dv}{Dt}(t,x) dx = F_{V_t}\:.$$
Remark. Dealing with Hamiltonian mechanics, there is a similar conservation law concerning probability when one studies statistical ensembles, i.e. statistical (Hamiltonian) mechanics. In that
situation, in fact, Liouville's theorem establishes that the Hamiltonian evolution preserves the canonical volume of the space of phases. In other words (6)' holds true, where $v$ is the field of Hamiltonian velocities $(dq/dt,dp/dq)$. As a consequence the law of conservation of probabilities, described by the Liouville density $\rho$, can be stated in the simpler version:
$$\frac{D\rho}{Dt}=0\:,$$
which, in turn, adopting the standard notation of statistical Hamiltonian mechanics, reads:
$$\frac{d\rho}{dt}=0\:,$$
and, eventually, can be re-formulated into the celebrated Liouville equation form:
$$\frac{\partial \rho}{\partial t} + \{\rho , H\}=0\:.$$
Best Answer
The places in physics where commutation of partial derivatives tends to be important are in the identities of vector calculus. The situations where these identities might seem to break down is when there is some kind of topological winding. Then the partial derivatives commute at almost all points except some small set where they are undefined but still can be given some meaning as a delta function.
For instance consider the vector potential $\mathbf{A}$ and its relation to the magnetic field $$\mathbf{B}=\nabla\times\mathbf{A},$$ $$\nabla\cdot B = (\partial_x\partial_y -\partial_y\partial_x)A_z + (\partial_y\partial_z -\partial_z\partial_y)A_x + (\partial_z\partial_x -\partial_x\partial_z)A_y.$$ If the partial derivatives commute acting on $\mathbf{A}$ then the divergence of $\mathbf{B}$ vanishes, and there is no magnetic charge density. But suppose we want a theory with magnetic monopoles ---the commutation of partial derivatives needs to break down somewhere.
So one possibility might be to take the vector potential to be the continuous function appearing in Kyle Kanos's answer $$A_x=A_y=0$$$$A_z=\frac{xy(x^2-y^2)}{x^2+y^2},$$ Here the partial derivatives commute everywhere except the origin, where you get only a finite difference (not like a delta function). So this is interesting but not physically relevant since the Lebesgue integral of the magnetic charge density over any finite volume is still zero.
Instead the magnetic monopole is described by the vector potential of a Dirac string: $$A_x = \frac{\mp y}{r(r\pm z)}$$ $$A_y = \frac{\pm x}{r(r\pm z)}$$ $$A_z = 0,$$ where the two choices of sign are just related by a gauge transformation. Unlike the previous example this vector potential is not defined at the origin. If you work it out in spherical coordinates you will find that the divergence of the magnetic field is a delta function, so this does indeed describe a non-zero magnetic charge.
The fact that we need more than one gauge equivalent function, and the fact that the functions are not defined at all points are typical of the way the commutation of partial derivatives fails in physics.
Here is another example. Given a function $f(x,y)$ of two variables $$(\nabla\times \nabla f)_z = (\partial_x\partial_y -\partial_y\partial_x) f, $$ and so by Stoke's theorem if the partial derivatives commute the line integral of a gradient vector field around a closed loop is zero.
Now take the function $\phi(x,y)$ which just returns the angle from $0$ to $2\pi$. There is a discontinuity at the positive x-axis where $0$ meets $2\pi$, but the gradient of $\phi$ can still be defined continuously here. We might consider a second function $\phi^\prime$ which instead returns the angle in the range $-\pi$ to $3\pi/4$ shifting the discontinuity to the negative y-axis. This function has the same gradient as $\phi$ and is like the additional gauge equivalent vector potential in the Dirac string example above.
If we look at how to take gradients and curls in cylindrical coordinates $$\nabla\phi = \nabla\phi^\prime = \rho^{-1}\hat{\phi},$$ where $\rho = \sqrt{x^2+y^2}$ and $\hat{\phi}$ is the unit vector in the angular direction. Taking the curl, $$\nabla\times(\nabla\phi)=-\partial_z (\rho^{-1}) \hat{\rho} + \rho^{-1}\partial_\rho (\rho\,\rho^{-1})\hat{z} = 0. $$
But even though the curl appears to be zero, clearly the line integral of either $\phi$ or $\phi^\prime$ around a closed loop containing the origin is $2\pi$, which seems to violate Stoke's theorem. However in any gauge the angle $\phi$ and its gradient are not defined at the origin, and that is where the commutivity of partial derivatives breaks down. Since we know the line integral of any closed loop around the origin is $2\pi$ this means $$(\nabla\times\nabla\phi)_z = (\partial_x\partial_y -\partial_y\partial_x) \phi = 2\pi\delta(x,y).$$
This may seem to be physically irrelevant, but in superfluids the function $\phi$ is the order parameter, and its gradient is the superfluid velocity. The superfluid is only allowed to have non-zero vorticity (curl of the velocity) in the core of a topological defect. In the limit of zero thickness, the topological defect is just like the discontinuity at the origin in the example above.