My interpretation is that you are asking why one should care for the density function of a random variable $X$, when what we use for simple probability calculations is the cumulative distribution function (the integral of the density function).
There are many calculations in which we use the density function. For example, if $X$ has density function $f_X(x)$, then $E(X)=\displaystyle\int_{-\infty}^\infty xf_X(x)\,dx$. More generally, if $Y=g(X)$, then $E(Y)=\displaystyle\int_{-\infty}^\infty g(x)f_X(x)\,dx$.
Remark: Your interpretation of the density function as rate of change of probability is good. Similarly, acceleration is rate of change of velocity. Even if we are only interested in velocity, acceleration is a useful notion.
I would agree with you that cumulative distribution function is more fundamental than density. However, that foes not mean that density of no importance. In the case of the normal, we have the additional fact that while density is an "elementary" function, the cdf is not.
As A.S.'s comment indicates, both distributions relate to the same kind of process (a Poisson process), but they govern different aspects: The Poisson distribution governs how many events happen in a given period of time, and the exponential distribution governs how much time elapses between consecutive events.
By way of analogy, suppose that we have a different process, in which events occur exactly every $10$ seconds. Then the number of events that happen in a minute (i.e., $60$ seconds) is deterministically $6$, and the amount of time that elapses between consecutive events is, of course, deterministically $10$ seconds.
In contrast, in a Poisson process with a mean rate of one event every $10$ seconds (i.e., $\lambda = 1/10$), the number of events that happen in a minute is not deterministically $6$, but it has a mean of $6$. The exact distribution is given by the Poisson distribution:
$$
P_k(t) = \frac{(\lambda t)^k}{k!} e^{-\lambda t}
$$
where $t = 60$ seconds is the time window. Thus, the probability that no events occur in a minute is given by
$$
P_0(t) = \frac{(6)^0}{0!} e^{-6} = e^{-6} \doteq 0.0024788
$$
whereas the probability that $6$ events occur in a minute is given by
$$
P_6(t) = \frac{(6)^6}{6!} e^{-6} = \frac{46656}{720} e^{-6} \doteq 0.16062
$$
That is obviously much more likely, as you would expect.
Similarly, the time between events is also not deterministically $10$ seconds, but it has a mean of $10$ seconds. The actual time distribution is the exponential distribution, which can be specified using its CDF (cumulative distribution function)
$$
F_T(t) \equiv P(T < t) = 1-e^{-\lambda t}
$$
The CDF provides essentially the same information as the PDF (probability density function), whose formulation you gave in your question; in fact, the derivative of the CDF is the PDF. However, the CDF is sometimes easier to understand intuitively, so I'll explain using the CDF here.
In this case, the probability that the time between events is less than $10$ seconds is $F_T(10) = 1-e^{-1} \doteq 0.63212$, whereas the probability that the time between events is less than $60$ seconds is $F_T(60) = 1-e^{-6} \doteq 0.99752$. The probability that it is greater than $60$ seconds is $1-F_T(60) = e^{-6} \doteq 0.0024788$, and you'll notice this is equal to the probability that no events occur in a given minute. This is no coincidence; in a Poisson process, which is memoryless, the probability that the time between events is greater than a minute is naturally equal to the probability that no events occur in that minute!
Best Answer
For a continuous distribution, it is the median (not the mean) that satisfies the equation:
$$\mathbb{P}(X < x_\text{median}) = \mathbb{P}(X > x_\text{median}).$$
This does not hold exactly for a discrete distribution, owing to the fact that there is positive probability at the point $x_\text{median}$, but still, it should hold approximately. Now, the Poisson distribution is a positively skewed distribution, and its mean is higher than its median. Thus, the result you are observing is quite unsurprising.