Certainly the idea of the proof is correct. What you want to end up showing is that for all $t \in \mathbb{R}$, and for all $x \in \mathbb{R^2}$, we have
\begin{align}
h(t) = t f(x). \tag{*}
\end{align}
However, this requires some case work, which I'll outline for you, but I'll let you fill in the details.
Case $1$: $x = 0$.
(You supply the proof of why (*) is true)
Case $2$: $x \neq 0$. Here we have $3$ subcases:
- $t> 0$
- $t =0 $
- $t < 0$ (here you need to make use of the given property of $g$)
Once you do this, then, you will have shown that for all $x \in \mathbb{R^2}$, $h: \Bbb{R} \to \Bbb{R}$ is a linear transformation. By the way, there are a few mistakes in your definition of differentiability. I'm not sure if these are typos or conceptual mistakes, but I'll highlight them anyway. First, you wrote
A function $f: \Bbb{R} \to \Bbb{R}$ is differentiable at a point $a \in \Bbb{R}$ if there exists a linear transformation $λ(h): \Bbb{R} \to \Bbb{R}$ such that ...
It should just say $\lambda : \Bbb{R} \to \Bbb{R}$, NOT $\lambda(h)$. $\lambda$ is the linear transformation, while $\lambda(h)$ is the real number you get when you evaluate $\lambda$ at a point $h$ in its domain. This kind of mistake appears again. You wrote:
Thus $h$ is differentiable if for all $t \in \Bbb{R}$, there exists a transformation $Dh(k): \Bbb{R} \to \Bbb{R}$ such that
\begin{equation}
\lim_{k \to 0} \dfrac{h(t+k) - h(t) - Dh(k)}{k} = 0.
\end{equation}
The correct statement is: $h$ is differentiable if for every $t \in \mathbb{R}$, there is a linear transformation $Dh(t) : \mathbb{R} \to \mathbb{R}$, such that
\begin{equation}
\lim_{k \to 0} \dfrac{h(t+k) - h(t) - Dh(t)(k)}{k} = 0
\end{equation}
Notice that the derivative at $t$ is denoted $Dh(t)$. This itself is a linear transformation from $\Bbb{R}$ into $\Bbb{R}$. So you have to evaluate this on $k$.
I suggest you read the definition of differentiability again, and pattern match what kind of object everything is.
Also, the correct answer for the derivative of $h$ is: for every $t \in \Bbb{R}$, $Dh(t)(\cdot) = h(\cdot)$, because $h$ is a linear transformation, as shown by equation (*) (I'd suggest you take a look at Theorem $2$-$3$ in the next section)
Added:
In this special case, as mentioned in the comments, the domain and target space of $h$ are $\mathbb{R}$. So you can use the old (single variable) definition of differentiability. You should also try to see why existence of $h'(t)$ (using single variable definition) and existence of $Dh(t)$ (using the definition above) are equivalent, and prove that $h'(t) = \left( Dh(t) \right)(1)$. (This is true in general for any function $h: \mathbb{R} \to \Bbb{R}$, not just the one defined in this question)
A different solution is the following 'lazy exponential' - there are easier solutions (maybe look up bump functions), but I like delay ODEs. set
\begin{align}x\in(-\infty,0]&\implies f(x):=1,\\
x\in(0,1] &\implies f(x) := 1+x, \\
x\in (1,2] &\implies f(x) := 1+x + \frac{(x-1)^2}{2!},\\
x\in(2,3] & \implies f(x) := 1+x + \frac{(x-1)^2}{2!} + \frac{(x-2)^3}{3!},
\end{align}
and in general
$$x\in(n,n+1]\implies f(x) := \sum_{k=0}^{n+1} \frac{(x-k+1)^k}{k!}. $$
If you differentiate, you find for $x\in (n,n+1)$, where $n>1$:
$$ f'(x) = \sum_{k=1}^{n+1} \frac{(x-k+1)^{k-1}}{(k-1)!} =\sum_{j=0}^{n} \frac{(x-j)^{j}}{j!}= f(x-1)$$
so to the right of 1, it solves a delay ODE with initial data prescribed on $x\in(0,1]$ above. $f'$ is clearly discontinuous at $0$, but $$\left.\frac{d}{dx}\frac{(x-1)^2}{2!}\right|_{x=1} = 0 $$ so the derivative is continuous at $x=1$. In general, for any integer $n\ge 2$, near $x=n-1$, all the terms $\frac{(x-h+1)^h}{h!}$ for $h<n$ are smooth, and the newly added term $T_n$,
$$ T_n(x) := \begin{cases} \frac{(x-n+1)^n}{n!} & x>n-1,\\ 0 & x\le n-1\end{cases}$$
is $C^1$. Conclusion - $$f \in C^0(\mathbb R)\cap C^1(\mathbb R\setminus \{0\}).$$
To finish, we use the delay ODE, which says that differentiating is the same as translating the function to the right by one. Thus
for $x\in \mathbb (0,\infty)\setminus \mathbb N$, $i\in\mathbb N$,
$$ f^{(i+1)}(x+i) = f'(x).$$
So the discontinuity of $f^{(i+1)}$ at $x=i-1$, and the continuity at integers $x=\tilde i > i-1$ follows directly from the dis/continuity of $f'$ at $0,1,2,\dots$. We conclude
$$ f \in C^0(\mathbb R)\cap \left(\bigcap_{k=1}^\infty C^k(\mathbb R\setminus{\{0,1,\dots,k-1\}})\right).$$
Best Answer
$\newcommand{\R}{\mathbb{R}}$ To long for a comment but not really an answer (also relates to this discussing):
I haven't read the text by Spivak. But since he is mentioning manifolds in the title let me stress this point: When it comes to considering the $\mathbb{\R}^n$ as a canonical manifold basically every abstract concept necessary for manifolds becomes the "simpler" version known from multivariable calculus. So in that setting it is only good(!) that concepts are the same, because really they were developed to be exactly that. But, the definition of differential you give with the difference quotient can not be adapted for abstract manifolds, so we need a knew concept. And only in case of a scalar (!) function $f : \R^n \to \mathbb{R}$ can one identify the differential $Df(a)$ with the $1$-form $df(a)$.
Is this addressing your problem add all?
EDIT
Okay, now I had a look into the book by Spivak. And here is what I think he is doing:
Define for simplicity differential forms first on $\R^n$. Thereby you can always check that this concept really is just a different view on what is know as vector calculus, I guess.
But with this amazing advantage that everything he is developing with differential forms can in the next chapter be easily applied also for manifolds! Which he will introduce in the next chapter.
And to address your point, why having two definitions for the same thing? Because to me this is one of the mile stones in every theory I would say, when seeing that under a special situation this concept actually coincides with an already established concept. Just keep studying this stuff and just worship how elegant for example the fundamental theorem of calculus in any (!!!) dimensions turns out to be, when formulated in differential forms.