As you correctly stated at the beginning, the units of $f'(x)$ are easily seen from writing the derivative as
$$ f'(x) = \frac{df}{dx} $$
so the units are the same as units of $f/x$. However, your concerns about the temperature can't be justified. The units of absolute temperature and the temperature difference are the same. In particular, the international system of units, SI, uses 1 kelvin for both.
One may also work with temperature scales that are not absolute, e.g. the Celsius degrees. The absolute zero doesn't correspond to 0 °C; in this sense, these scales are "nonlinear". However, a Celsius degree is still a unit of temperature as well as temperature difference. As a unit of temperature difference, 1 °C and 1 K are the very same thing. You can surely never forget or omit units such as degrees (of temperature) from physical quantities, whether they are computed as derivatives or not.
Similarly, adding two absolute temperatures is invalid.
It may be unnatural or useless in most physical situations (see Feynman's "Judging Books By Their Cover") but it is a valid procedure when it comes to the units. After all, absolute temperatures are just energies per degree of freedom so adding absolute temperatures isn't much different from adding energies which is clearly OK.
In particular it does not make sense to do $f'(x)\cdot y$ where $y$ is an absolute temperature.
It makes perfect sense. Thermodynamics is full of such expressions. For example consider $f(x)=S(t)$, the entropy as a function of time. Then $S'(t)\cdot T$ is a term that appears in the rate of change of some energy according to the first law of thermodynamics.
Quite generally, it is not sensible to single out temperature in these discussions. The same comments hold for distances, times, or pretty much any other physical quantities. Take time. One may consider the "current year". It's some quantity whose unit is 1 year. (Similarly, the position of something in meters.) And one may consider durations of some events whose units may also be years. It is the same unit. It is obvious that the difference $A-B$ i.e. any difference has the same units as $A$ as well as $B$. In my "current year" analogy, $Y=0$ corresponds to the birth of Jesus Christ, a random moment in the history of the Universe. That's totally analogous to $t=0$ °C, the melting point of ice. But in both cases, the time differences or temperature differences have the same units as the quantities from which the differences were calculated – such as temperature (whether they're absolute or not) or dates.
It wouldn't make any sense to have different units for quantities and their differences because dimensional analysis would cease to hold: one could no longer say, among other things, that the units of $A-B$ are the same as units of $A$ or $B$ separately.
It is very correct that one cannot calculate a sensible value of $\exp(a)$ if $a$ is dimensionful i.e. if it has some nontrivial units. Such an exponential would be adding apples and oranges, literally. Express $\exp(a)$ as the Taylor expansion, $1+a+a^2/2+a^3/6+\dots$. If $a$ fails to be dimensionless, each term has different units so it's not dimensionally correct to add them. For this reason, all exponentials (and, with a somewhat greater tolerance, logarithms) in physics are exponentials of dimensionless quantities (which have no units). The desire to avoid physically (and mathematically) meaningless quantities such as exponentials of dimensionful quantities is one of the very reasons why we use units and dimensional analysis at all. It is not a "problem"; it is a virtue and the very point of these methods.
The NIST style guide is pretty good — that's a place where people really care about getting details right.
I use lower-case names for spelled-out units, even when named for famous people or having uppercase abbreviations (N -> newton, J -> joule, L -> liter (unless you count $\ell$ for liter), K -> kelvin). I think that "degrees kelvin" is entirely by analogy with "degrees Celsius" and "degrees Fahrenheit" (the latter two of which I think I have always seen capitalized).
I don't pluralize "kelvin" when talking about temperatures: I talk about temperatures like "two hundred fifty millikelvin," or "four kelvin" for the boiling point of helium, or "three hundred kelvin" for room temperature. This is not consistent with the way that I would discuss a length unit, or a mass unit.
I think that this may be because temperature is an intensive variable. If I have a thing that weighs a kilogram, and another thing that weighs four kilograms, and I put them together, I know that I have five kilograms worth of stuff. But if I have some stuff at one kelvin, and some other stuff at four kelvins, and I mix them together, I don't get some stuff at five kelvins. I know this, and so I don't think of "a kelvin" as a lump of temperature that I can carry around and add or subtract to things.
I feel the same way about the hertz: I have no desire to say "sixty hertzes." Combining an oscillator at 60 Hz and an oscillator at 10 Hz gives me something much more complicated than an oscillator at 70 Hz. I notice that "hertz" is listed as one of your three exceptions, though.
If that's really my thought process I would make the same decision about the pascal (for pressure) and the poise (for viscosity); I can't think off the top of my head of another intensive quantity with a named unit. I think that if you asked me I would tell you that air pressure at sea level is "ten to the five pascal," but I'm focusing too hard on it and I'm honestly not sure.
In response to a comment: I definitely
do say things like "two atmospheres of pressure," but never "two bars" or "one thousand torrs." It could well be that dealing with kelvins one by one is so rare that I don't think of them as being countable. Interesting.
Best Answer
Let's focus on two example equations where temperature enters. The first example is Newton's law of cooling: $$ J = c (T_h-T_c) $$ where $J$ is the heat flux from a hot reservoir at temperature $T_h$ and a cold reservoir at temperature $T_c$ and $c$ is some constant that depends upon the interface details. It should be clear that in this equation we could (if we wished) write instead: $$ J = c\left((T_h-T_r)-(T_c-T_r)\right)$$ and insert some reference temperature $T_r$. Thus this equation makes sense whether or not we use absolute temperature (measured in Kelvin) or relative temperature (measured in degree Celsius or Fahrenheit, and which has precisely such a reference temperature).
Instead consider the Stefan-Boltzmann equation: $$ P = A\sigma(T_h^4-T_c^4) $$ because $x^4$ is a non-linear function, we can no longer do the same shifting trick. Thus, the Stefan-Boltzmann law can only hold in absolute units. This should make it clear that any quantity where the units include Kelvin raised to any non-linear power can't make much sense when written in Celsius.