It's not known that modern set theory is consistent; in fact, by the Incompleteness Theorem, we can't ever have a system of axioms that we can prove is consistent. Which means that the only condition we can rely on for determining whether a set of axioms is "right" is whether or not it produces absurd results.
Under $ZFC$, we have different sizes of infinity - there are sets which are larger than the set of natural numbers in a precise sense. We also have a lot of weirdness involving the Axiom of Choice - for example, with the Axiom of Choice, a theorem of Banach and Tarski states that a hollow sphere can be disassembled into five pieces and then reassembled (without stretching, tearing, or otherwise deforming the pieces) into two spheres that are both identical to the first one in both size and shape. But the Axiom of Choice simply states that given a set of sets, we can "choose" one element from each set - which seems intuitively true.
A finitist's perspective on $ZFC$ is often that results like the hierarchy of infinite cardinals and the Banach-Tarski paradox are absurd - that they should count as contradictions, because they patently disagree with the intuitive picture of mathematics. The sensible conclusion is that one of the axioms of $ZFC$ is wrong. Most of them are intuitively obvious, because we can demonstrate them with finite sets - the only one we can't is Infinity, which states that there exists an infinite set. So a finitist's conclusion is to reject the Axiom of Infinity. Without that axiom, $ZFC$ becomes purely finitistic.
Now, many finitists are happy to stop here. But some are bothered by the fact that we still have an infinite collection of natural numbers; the infinite still "exists", in a sense, and gives the opportunity for the above weirdnesses to arise in the same way. So some people (including some mathematicians) subscribe to ultrafinitism and insist that there are only finitely many numbers at all. One ultrafinitist mathematician I know defines the largest integer to be the largest integer that will ever be referenced by humans.
Among mathematicians, ultrafinitists are much rarer than simple finitists. Finitists generally agree with you that "unboundedness" is a natural idea - it's essential, for example, in the definition of a limit. But they would go on to insist that this is just a formalism - that a limit, for example, is just a statement of eventual behavior, involving only finite numbers. So $\infty$ isn't an object, it's just a shorthand. This is (at least to my mind) more mathematically defensible than ultrafinitism.
EDIT: Since a lot of people seem to be having a hard time with my first sentence, I thought I'd clarify. The Incompleteness Theorem states that we cannot have a set of axioms powerful enough to express arithmetic and still be able to prove its consistency within the system. The reason I didn't include this phrase above is because it unnecessarily weakens the point. Any axiom system intended to codify all of mathematics defines the idea of "proof"; if, say, $T$ is intended to underlie all of mathematics, then by "proof" we must mean "proof in $T$". With such a $T$, we can say that $T$ cannot be proven consistent at all; because by Incompleteness, any proof of the consistency of $T$ would not be a proof from inside $T$, but $T$ is supposed to be powerful enough that all proofs are proofs from inside $T$. Thus: we can't ever have a system of axioms for mathematics that we can prove is consistent - full stop.
The notion of "countably infinite" is well named. Another word you can use is "enumerable," which is even more descriptive in my opinion.
I understand your intuition on the subject and I see where you're coming from. Let me try to give you some insight into the agreements about infinity that have been reached over the years by the mathematicians who've worked on this problem. (This is what I wish someone had explained to me.)
"Countably infinite" just means that you can define a sequence (an order) in which the elements can be listed. (Such that every element is listed exactly once.)
The natural numbers are the most naturally "countable"—they're even called "counting numbers"—because they are the most basic, natural sequence. The word "sequence" itself is wrapped up in what we mean by natural numbers, which is just one thing following another, and the next one coming after that, and the next one after that, and so on in sequence.
But the integers are countable as well. In other words, they are enumerable. You can specify an order which lists every integer exactly once and doesn't miss any of them. (Actually it doesn't matter, for the meaning of "countable," if a particular element gets listed more than once, because you could always just skip it any time it appears after the first time.)
Here is an example of how you can enumerate (count, list out) every single integer without missing any:
$0, 1, -1, 2, -2, 3, -3...$
The rational numbers are also countable, again because you can define a sequence which lists every rational number. The fact that they are listed means (at the same time) that they are listed in a sequence, which means that you can assign a counting number to each one.
The real numbers are not countable. This is because it is impossible to define a list or method or sequence that will list every single real number. It's not just difficult; it's actually impossible. See "Cantor's diagonal argument."
This will hopefully give you a solid starting point to understanding anything else about infinite sets which you care to examine. :)
Best Answer
You are confusing numbers with numerals. Numerals are symbols that represent numbers. Numbers do not have any intrinsic representation as sequences of digits or anything else. Instead, we devise different schemes for representing numbers with numerals. For example, in one scheme, we use sequences of digits
0,1,2,3,4,5,6,7,8,9to represent certain numbers; the numeral119represents a certain number. But there is nothing privileged or special about this numeral; in a different, similar system, the same number is represented with the numeral1110111; in a different, less similar system the same number is represented with the numeral百十九, in another system it is represented with the numeralCXIX, in a different system it is represented with the numeralone hundred and nineteen, and in a different system again it is represented with a certain pattern of electron flow in a chunk of silicon.So the question of whether a certain number "has digits in it" is a category error. Numbers never have digits. Some systems of numeration use digits, and numerals in those systems have digits in them. But the number of digits will depend on which system you are using.
119is a three digit numeral, and1110111is a seven-digit numeral, but they both represent the same number.The question that does make sense to ask is whether a certain system of numerals can represent a certain number. For example, some systems are able to represent the number one-half. One might write it in one system as $\frac12$, and in another system as
0.5. Some systems simply have no representation for one-half.So we can ask if the standard decimal system, the one which uses digits
0,1,2,3,4,5,6,7,8,9, has a representation of the number infinity, and if so how many digits are used to represent it. And the answer is no, as usually understood, this system has no representation for the number infinity. (Or, more precisely, for any of the several numbers called "infinity".)