I have to admit that this is not really an answer, but rather some sort of meta-answer with some very general remarks which I hope do not bore everyone reading this; it just seems to me that this is necessary to indicate that it is rather misguided, as Yemon already says in the comments and I strongly agree with, to ask such a question if some book introduces elementary number by means of category theory.
Mathematics is all about the nontrivial, unexpected relationships. Category Theory is not really about finding such relationships, but rather about the correct setting, language and color some theory is developed. This point of view does not really contradict the hitherto development of category theory into a huge area of mathematics in its own right, full of nontrivial deep theorems; namely because often there is some geometric or whatever background which is our real motiviation. There are ubiquitous examples (model categories, topoi, stacks, $\infty$-categories, ...) which I don't want to elaborate here.
Anyway, as I said, mathematics really starts when something unexpected happens, which does not follow from general category theory. For example, the covariant functor $\hom(X,-)$ is always continuous, but when is it also cocontinuous, or respects at least filtered colimits? It turns out that this leads to a natural finiteness condition on $X$, namely we call $X$ then finitely presented. But finally to arrive at the question, $\mathbb{Z}$ is easily seen to be a inital object in the category of rings, but what theorems from category theory are known about initial objects? Well there is nothing to say, expect that every two initial objects are canonical isomorphic, which is just a trivial consequence of the definition. So $\hom(\mathbb{Z},-)$ is easy to describe, but what about the contravariant functor $\hom(-,\mathbb{Z})$? What happens when you plug in $\mathbb{Z}[x,y,z]/(x^n+y^n=z^n)$ for some fixed $n>2$? Does category theory help you to understand this? This example also shows that although the Yoneda-Lemma says that an object $X$ of a category is determined by its functor $\hom(X,-)$, it does not say you anything about the relationship of $X$ with other objects, for example when we just reverse the arrows. Instead, we have to use a specific incarnation of the category and its objects in order derive something which was not there just by abstract nonsense.
Perhaps related questions are more interesting: Which investigations in elementary number theory have led to some category theory (for example, via categorification), which was then applied to other categories as well, thus establishing nontrivial analogies? Or for the other direction, which general concepts become interesting in elementary number theory after some process of decategorification? But in any case, it should be understood that you have to digest elementary number theory before that ...
There is an interaction between category theory and graph theory in
F.~W.~Lawvere. Qualitative distinctions between some toposes of generalized graphs. In {\em Categories in computer science and logic (Boulder, CO, 1987)/}, volume~92 of {\em Contemp. Math./}, 261--299. Amer. Math. Soc., Providence, RI (1989).
which we have exploited in
R. Brown, I. Morris, J. Shrimpton and C.D. Wensley, `Graphs of Morphisms of Graphs', Electronic Journal of Combinatorics, A1 of Volume 15(1), 2008. 1-28.
But that is actually about possible categories of graphs, which may be the opposite of the question you ask.
If you look at groupoid theory, then "underlying graphs" are fundamental, for example in defining free groupoids. See for example
Higgins, P.~J. Notes on categories and groupoids, Mathematical Studies, Volume~32. Van Nostrand Reinhold Co. London (1971); Reprints in Theory and Applications of Categories, No. 7 (2005) pp 1-195.
Groupoids are kind of "group theory + graphs".
Best Answer
A few points:
The category of graphs is certainly not equivalent to the category of categories. But they are related (for more on that see (3)).
As you allude to, there are multiple categories of graphs to be interested in. I've had a go at naming some of them here, including directed/undirected, multi/simple, and various conditions on loops, trying to view them from a common framework as being certain categories of presheaves.
Let $Gph$ be the category of directed multigraphs (as usual for category theorists), and let $U: Cat \to Gph$ be the forgetful functor. As you point out, $U$ has a left adjoint $F: Gph \to Cat$, which sends a graph $\Gamma$ to the category of "paths" in $\Gamma$. This adjunction is even monadic, exhibiting $Cat$ as a category of algebras over $Gph$. In this way, it's reasonable to regard a category as a graph with extra structure. I'd hazard a guess that dually the functor $F: Gph \to Cat$ is maybe comonadic? (EDIT: Yes, I'm pretty sure that $F$ preserves all equalizers. It is clearly a conservative left adjoint between complete categories, so it's comonadic by the dual of the crude monadicity theorem.) In this way, it should be reasonable to regard a graph as a category with extra structure associated to being a free category. This would give some more justification for your approach of viewing a graph as a category with extra structure.