I am surprised both by the approach of your textbook (you don't need determinants to introduce the distinction between singlar and non-singular matrices, nor to solve linear systems), and by the fact that you qualify this approach as abstract. I would qualify a don't-ask-questions-just-compute attitude as concrete rather than abstract. Maybe you use "abstract" to mean "hard to grasp", but it is not the same thing; for me often the things hardest to grasp are complicated but very concrete systems (in biochemistry for instance). In mathematics (and elsewhere, I suppose) it is often asking conceptual questions that leads to abstraction, and I sense that what you would like is a more conceptual, and therefore more abstract approach.
But abstraction is present in many fields of mathematics, like linear algebra, for a more improtant reason as well, namely for the sake of economy and generality. Linear algebra arose as a set of common techniques that apply to problems in very diverse areas of mathematics, and only by an abstract formulation can one express them in such a way that they can be applied whereever needed, without having to reformulate them in each concrete situation. It would be motivating to have seen at least one such concrete application area before entering the abstraction of the subject, and I think that would be a sound approach. However this would involve introducing many details that in the end are independent of the methods of linear algebra, and I guess there is often just not the time to go into such preparations.
So to answer your questions.
Linear algebra is an abstract subject, so it should not surprise tht freshmen feel it is so. But it is not abstract because of determinants, which are just a concrete tool that allows certain things to be expressed more explicitly than without them. Saying a linear map is invertible is a more abstract formulation then saying $\det A\neq0$ where $A$ is a matrix of that linear map in some basis.
Yes, geometric insight helps understanding linear algebra, and you should have some geomtric intuition for notions as subspaces, span, kernels, images, eigenvalues. But determinants are somewhat different; while you certainly should have some geometric intuition for determinants in terms of volume when doing calculus, there is not much to gain from this in purely algebraic situations, and in fact I would know no geometric interpretation at all of the determinant of a complex matrix, or of the determinant that defines the characterisitic polynomial.
To understand linear algebra better, you should try to go beyond concrete computational questions, and try obtain a more conceptual understanding of what is being done.
As for the mysteries of determinants, you may want to have a deeper understanding than just that they exists and magically solve certain problems (like determining which square matrices are invertible). For that I would refer to this question.
Abstract algebra has an interesting way of making a problem more transparent by forgetting about superfluous properties. I'll give some real world applications to illustrate:
Let's say you're a physicist studying the motion of a moving particle modeled by some differential equation. To find solutions to such an equation, one typically takes the Fourier transform and solves a corresponding algebraic problem. The Fourier transform is essentially allowing us to see past the complexity of arising from taking derivatives to illuminate an underlying algebraic problem.
As another example, lets say you are studying the effects of a gravitational field in a certain area of spacetime. Particles which travel through this area are subject to the curvature of spacetime induced by the gravitational field. Again this situation is extremely complicated. However, We can reduce the problem to an algebraic problem locally. That is, we use the tangent bundle and a smoothly varying metric to describe its motion.
Another example dates all the way back to Descartes. Geometric shapes are hard to understand, but imposing coordinates on such objects allows us to use algebraic techniques to understand the object better.
Anomolies in physics arise as elements of cohomology groups. Without the notion of a group, they would be very hard to calculate. You may not even know where to start.
The bottom line is, we know how to do algebra. It is the ability to translate of difficult problems into algebraic ones that makes it so useful.
Best Answer
You have never seen it used for anything?
The fact that $A\text{adj}(A)=\text{adj}(A)A=\det(A)I$ is the standard one-half of the proof that $A\in\text{GL}_n(R)$ iff $\det(A)\in R^\times$, where $R^\times$ is the group of units of $R$.
The conceptual meaning of the adjugate matrix is somewhat complicated. Really, you can imagine it as being the adjoint of $\bigwedge^{n-1} A$ with respect to a somewhat natural pairing. More information can be found here.