I read a set of notes recently (unfortunately I can't find the link) in which the author made a statement of the form "differentiation of a function with respect to a function doesn't make sense". By this, do they simply mean that in "ordinary" calculus taking a derivative of a function with respect to another function, where we consider the whole range of the function simultaneously, has no meaning? Obviously, in variational calculus one has functionals which do depend on the entire function and hence a functional derivative is with respect to a function where we consider it as a whole. But does this have a meaning in elementary calculus?
After reading this statement I haven't been able to shake a nagging doubt in my mind that I may be misunderstanding things. I mean, in "ordinary" calculus, the only case that I can think of where we take a derivative of function with respect to another is in the chain rule. However, here aren't we doing a pointwise procedure? For example, if one has a function $$g:x\mapsto g(x)=y$$ and another function $$f:y\mapsto f(y)=z$$ Then, $$f\circ g:x\mapsto f\circ g(x)=f(g(x))=f(y)$$ and this is done pointwise, i.e. for each value $g(x)=y$ we "plug" this into $f$ and obtain an output value $f(y)=f(g(x))$. Then taking the derivative is also carried out in a pointwise fashion, $$\frac{d(f\circ g)(x)}{dx}=f'(g(x))g'(x)$$ so in this sense one is treating the output of $g$ as an input variable for $f$ and then taking a derivative of $f$ with respect to this variable (I think this is what is meant by $f'(g(x))$?)
If someone could clear this issue up for me I'd much appreciate it.
Best Answer
As you said in your post, the closest example I can think of where we take a derivative of a function wrt another function is the functional derivative:
$$\frac{\delta J}{\delta f(x)} = \frac{\partial \Phi}{\partial f}-\frac{d}{dx}\frac{\partial \Phi}{\partial f'}$$
Where:
$$J[f] = \int_a^b \Phi[f] dx$$
This derivative is defined in a vector space of functions $L$ for some given functional $J[f]: L \to \mathbb{R}$ for $f\in L$.
The result of the functional derivative is still a function of $x$, as can be seen by the several examples on the Wiki page. This is similar to an elementary derivative, where:
$$\frac{d}{dx} x^2 = 2x$$
So, one could argue that even in variational calculus, we are doing something very much like the chain rule in spirit.
Therefore, without actually seeing the context of your notes, I'm afraid its hard to say conclusively what they meant. However, if we restrict ourselves to elementary calculus, then the definition of the derivative does not support the notion of a derivative wrt some function $g$, qua function.
This is because the values of $g$ do not form an ordered field in the same sense that $h \in \mathbb{R}$ do for the definition of an elementary derivative:
$$\lim_{h\to 0} \frac{f(x+h)-f(x)}{h}$$
Even if we allow $g$ to be well-behaved wrt $x$ (i.e., smooth, continuous), the statement $g\to 0$ does not necessarily identify a unique limiting sequence of values for $f(x+g)-f(x)$, and we need that as a key part of our definition of the derivative. For example:
$$\lim_{g\to 0} \frac{f(x+g)-f(x)}{g}$$
For some function $g(x)$ is not well defined, since we have not specified how $g\to 0$. We don't have this problem when we are taking the limit wrt a real number $h\in \mathbb{R}$
Anyway, that is the closest I can come to making sense of such a statement. It would, of course, be helpful if you had the actual notes, but in the interest of helping you move past your concerns, I think that the author was not considering functional analysis in that statement.
As an aside, mathematics is such a huge field that any such sweeping, absolute statement is bound have a counterexample somewhere. For example, one can say it "doesn't make sense" to take the $\sqrt{2}$-th derivative of a function, and you'd be correct if you are talking ordinary calculus, but it would be wrong if we said that the concept, unconditionally, had no meaning, because we have fractional calculus.