Number line of real numbers can be represented as a 1-dimensional line of typically horizontal axis with zero being the origin. In this kind of representation, imaginary numbers would typically reside on the vertical axis, reaching to infinity in negative and positive values from origin, complex numbers residing on the plane with a non-zero distance to both of the axis.
Tried to searchengine, but didn't find such an abstract extension to be mentioned, which would give an absolute value of number/symbol to be LESS than zero?
I don't know if it would ever be useful, but neither was imaginary numbers when they were introduced to math. Nowadays we know there is lot of applications in electronics and quantum mechanics for example.
EDIT: As it turns out in the comments, imaginary numbers were already "under order" for practical applications. Practicality is not for now a point of interest personally, just curious whether field of mathematics has a name for such concept as described. Like, say I'm writing a story, where there is a creature stating it occupies a volume less than zero, which is not a negative volume, but "under zero" or "not even zero" .. And because of Reasons, I wanted to check whether such a system has a name in known maths.
Best Answer
A concept that comes close to what you describe in the question pre-edit (although I think the edit is the best part) is that of a negative definite quadratic form.
The generalization of the absolute value on the one-dimensional line to higher dimensional spaces is a norm. To compute the norm of a point you compute the squares of all the coordinates and add them. Now roughly speaking a quadratic form is a generalization of this concept where you take the squares of coordinates and instead of ordinarily adding them you first multiply each of them with a pre-chosen number (positive or negative) and then add the outcomes.
Quadratic forms are of interest to a wide range of mathematical topics, and hence are studied 'on their own'. If you want to, from that perspective, recognize the norms among the more general zoo of quadratic forms, the crucial property that the norms have is that they take only non-negative values (with zero occurring only if started with feeding the point zero into the norm). This property that some quadratic forms have and others don't is called being positive definite.
Naturally there is a mirror symmetric property called being negative definite meaning taking only non-positive values, with zero occurring only if you use the zero vector as input. In some contexts the negative definite quadratic forms are more natural/interesting than the positive ones. Among all quadratic forms that are not norms they are the ones closest to norms I would say. And as I said: norms are the natural extension of absolute values to higher dimensions.
That said: quadratic forms don't have the interpretation of a volume, as computing volumes involves multiplying 3 numbers together rather than just two. So the post-edit version of your question might need a different answer.