Introduction:
In Special Relativity, when we say a quantity is an invariant, we mean that it is a Lorentz invariant. What this means is that when you apply the Lorentz transforms to any frame of reference, the magnitude of the Lorentz invariant does not change. The Lorentz invariant is a scalar, which means that it can only have a magnitude, and can carry no other information. Instead, if we take something like a vector (something which has magnitude but also contains information about direction), then the magnitude would still be the same, but the information about the direction changes, so it could not be an invariant.
Also, in Special Relativity we say that events occur in 'spacetime', which is just a 4-dimensional 'stage' for events, including 3 dimensions of space and 1 dimension of time. In ordinary Newtonian mechanics, we conversely use space and time separately, and things like momentum are described only in 3 dimensions.
Because of this fact, physical concepts, like momentum and velocity, are described not by 3-dimensional vectors (for eg. $p = (p_x + p_y + p_z)$), but rather by four dimensional vectors, called four-vectors. Four-vectors have four components: the first one of time and the other three of space.
One more thing: in any transformation, the dot product of a vector with itself will always be an invariant, whether you are talking about $3D$ space or $4D$ spacetime.
The dot product $\vec{V} . \vec{V}$, remains the same when you change your coordinate axes (dotted lines).
Before we start: Conventions:- We assume natural units ($c = 1$), and the $(+---)$ metric. We label the time component as $0$, while the space components are labelled $1, 2, 3$.
1. How to take dot products of four vectors:
Let's say we have a four vector $A^{\mu}$. Now, how do you take the dot product of this vector with itself? If instead you had a $2D$ vector (say, $\vec{V}$), the dot product would have been $\vec{V} . \vec{V} = V_x^2 + V_y^2$. Now for four vectors, the dot product would be: $$A_{\mu} A^{\mu}$$
Notice that we have $A_{\mu}$, which is not the vector we started with. How do we get this? Like this: $$A_{\mu} = g_{\mu \nu} A^{\nu}$$
where $g_{\mu \nu}$ is the metric tensor. In flat spacetime, we use the Minkowski metric $\eta_{\mu \nu}$, given by:
$$ g_{\mu \nu} =
\begin{pmatrix}
1 & 0 & 0 & 0 \\
0 & -1 & 0 & 0 \\
0 & 0 & -1 & 0 \\
0 & 0 & 0 & -1 \\
\end{pmatrix}
$$
This is what I meant by the (+---) in the convention.
So, $$A_{\mu}A^{\mu} = (A^{0})^{2} - (A^{1})^{2} - (A^{2})^{2} - (A^{3})^{2} = (A^{0})^{2} - (A)^{2}$$
where $A$ is the vector describe in the three dimensions of space only. We subtract the space components, because of the negatives (+---) convention (the + means that time components remain positive, while the three - means that space components get negative).
2. Why mass is an invariant.
In SR, momentum (like other quantities) is also described by a four-vector. Mathematically, $$P^{\mu} = (E, p_x, p_y, p_z)$$
where $E$ is the energy, and the other components are the $3D$ momentum. Now, there is another relation: $$E^2 = \vec{p}^2 + m^2$$ where $m$ is the mass and the $p$ is the momentum (not the four-vector though!). Now, taking the dot product of $P^{\mu}$ with itself, $$P_{\mu}P^{\mu} = (P^0)^2 - \vec{p} = E^2 - \vec{p}^2$$ But, $E^2 = \vec{p}^2 + m^2$, therefore $$E^2 - \vec{p}^2 = m^2$$ So we get that the mass squared is the dot product of the momentum four-vector with itself. Therefore, $m^2$ is invariant. Therefore, $m$ is also an invariant.
3. So what's the deal about relativistic mass?
When an object is at rest, it has no momentum. Therefore we get $$E = m$$ When you apply the Lorentz transformation on the energy you get $$E^{'} = \gamma E - \beta \gamma P^X$$ but the momentum is $0$, so you get (because $E = m$) $$E^{'} = \gamma m$$ This is just $$E = \gamma mc^2$$ where I have taken $c = 1$, remember? But, this doesn't look so elegant, so some people replaced $\gamma m = M$ and got the world's most famous equation: $$E = Mc^2$$ But, relativistic mass is just a mathematical trick. We showed that mass is an invariant. When you multiply that by $\gamma$, you get $m_{rel}$, but it's not real. Your mass does not increase or decrease depending on your velocity. Rather, your inertia increases as your speed increases, that is why it takes infinite energy to accelerate an object to the speed of light.
For further info:
About how we see mass today: Why is there a controversy on whether mass increases with speed?
When relativistic mass can be useful: http://math.ucr.edu/home/baez/physics/Relativity/SR/mass.html
More about four-vectors in relativity: http://hyperphysics.phy-astr.gsu.edu/hbase/Relativ/vec4.html
Some resources on vectors and tensors (generally, the field of linear algebra):
https://www.mathsisfun.com/algebra/vectors.html
https://www.khanacademy.org/math/algebra-home/alg-vectors
https://www.youtube.com/watch?v=f5liqUk0ZTw
And a brilliant introductory series on the topic by 3Blue1Brown:
https://www.youtube.com/watch?v=fNk_zzaMoSs&list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab
And of course; the Wikipedia page: https://en.wikipedia.org/wiki/Mass_in_special_relativity
The mechanism is the nature of time and space. Time dilation is certainly an aspect of this.
The usual introductory thought experiment shows that if there is to be a conserved quantity behaving like momentum then its transverse component is invariant between inertial reference frames. But we know that transverse velocity, $u_y=\frac {\Delta y}{\Delta t}$, is not invariant because although $\Delta y$ is invariant, $\Delta t$ is not. But if we replace $\Delta t$ by the invariant interval $\Delta \tau$, then we have an invariant kinematic quantity with the dimensions of velocity. It is easy to show that
$$m\frac{\Delta y}{\Delta \tau}=m\gamma(u)\frac{\Delta y}{\Delta t}=m\gamma(u)u_y.$$
Similar expressions must apply for the other momentum components.
The justification of this expression for momentum is not really this thought-experiment, beautiful though it is, but the coherence of the whole fabric of relativistic dynamics. For example, we find that a body's three relativistic momentum components and its energy are the components of a 4-vector with an invariant modulus (mass) exactly analogous to the 4-vector representing the space-time interval between two events.
We also find that, using this definition of momentum, we can express the Lorentz force law in a way that holds in every inertial frame:
$$q(\vec E +\vec v \times \vec B)=\frac d{dt}m\gamma(u) \vec u.$$
This is relevant to your statement: "At speed $0.0001c$, force applied is approximately proportional to acceleration. At speed $0.89c$, the same force applied changes the speed of the object very little." I suggest that you look at it a rather different way: it is not the rate of change of $u$ that a force (in the same direction as $\vec u$) is proportional to, but the rate of change of $\gamma(u) u$.
Best Answer
Mass is a concept that developed with Newtonian mechanics as an invariant additive quantity. It is correctly called the inertial mass, because it comes from the F=ma identification, the constant response to acceleration.
When special relativity came at first as a proposal and then validated by innumerable data, the E=mc^2 formula gave a handle to understanding the inertial mass , i.e the response to acceleration.
This is the mass that has to be used when trying to accelerate a space ship close to velocity of light, and it was impressive enough, because together with the quantum mechanical description of nuclei ( binding energy curve) it showed a connection of mass and energy.
So in the quantum and special relativity domain, mass is not a conserved quantity, only energy and momentum are conserved.
Clarity of mathematics shows that these are a fourvector, with an invariant to lorenz transformations value when added as four vectors.Rest masses are not additive, four vectors follow vector addition rules, so a system of particles has an invariant mass the length of the added four vectors.
For individual particles it is the "rest mass", at the limit of velocity equal to zero.
The relativistic mass is variable with velocity:
In particle physics the invariant/rest mass identifies one to one the particles in the elementary particles table . All other invariant masses in principle are derivable using the four vector algebra.
So we call relativistic mass the famous mass in the formula E=mc^2 as a respect for the historical development, and invariant/rest mass the four vector "lengths" to stress the special relativity equations.