There is a lot to address here. You might be getting at any of the following points:
The existence of $\Pi_1^0$ statements: A statement is called $\Pi_1^0$ if, if it were false, it could be disproven by a finite amount of data. Goldbach's conjecture, "every even number is the sum of two primes", is $\Pi_1^0$, because to disprove it you just have to give an even number $2n$, and a nontrivial factor of $2n-p$ for each prime $p < 2n$. A $\Pi_1^0$ statement, if undecidable, is always true. See Wikipedia for the rigorous definition and much more.
It is far from the case that all interesting problems in number theory are $\Pi_1^0$. Consider the twin prime conjecture, "there are infinitely many pairs of primes which differ by $2$." This could be undecidable in two different ways. Maybe it is true, but we can't prove it. Or, maybe there are no twin primes past $10^{10^{10}}$, and we can't prove that. Either way, no finite amount of data will settle the issue.
Goodstein's theorem is not $\Pi_1^0$: Contrary to your statement, a finite amount of data cannot disprove Goodstein's theorem. Suppose that you knew that the Goodstein sequence starting at $100$ did not terminate. How could you convince me o this with any finite computation?
Now, in fact, Goodstein's theorem is true, because induction up to $\epsilon_0$ is valid. See this MO question for more. But consider the Collatz conjecture. Like Goodstein's theorem, it says that a recursively defined sequence starting at any integer eventually terminates. It is perfectly possible for Collatz to be either undecidable and true or undecidable and false.
ZFC does prove more than PA: The proof of Goodstein's theorem can be formalized in ZFC. So can the intuitive proof that PA is consistent, namely, "of course it's consistent, it is a set of true statements about $\mathbb{Z}$!" So it is true that being undecidable in ZFC is a lot stronger than being so in PA.
A point of philosophy: Some mathematicians will defend the belief that all statements about $\mathbb{Z}$ are either true or false (though they may be unprovable from the current axioms) but this need not be true about sets. The idea here is that there is only one set of integers, but there are many equally good versions of set theory. Scott Aaronson defends this view here; JDH defends the "more than one set theory" here. (I don't know whether or not JDH thinks there is more than one version of $\mathbb{Z}$.)
Note that this is much stronger than the claim that all $\Pi_1^0$ statements are true or false. Scott, for example, presumably believes that the twin prime conjecture is either true or false, even though no finite amount of computation could ever settle it.
I have not thought enough about this question to form an opinion; it is ultimately a matter of philosophy, not math.
The usual way that we prove that each sentence $\phi$ is either true or false in a model $M$ is by induction on the structure of $\phi$. So we prove it first for the case where $\phi$ is an atomic formula, then we prove that the property is preserved when we build more complicated sentences with logical connectives and quantifiers.
However, if we say that a sentence is provably true we mean that there is some theory $T$ such that the sentence is provable in $T$. The theory is often part of the context of an argument. Usually, because of the incompleteness theorem, the theory $T$ will not be able to prove true every sentence that happens to hold in some arbitrary model of $T$. So for example, there are sentences of arithmetic (with quantifiers) that are true in the standard natural numbers, but are not provably true in Peano arithmetic. It is the case that every sentence in that language is either true or false in the standard model of arithmetic, but it is not the case that every sentence is provably true or provably false in Peano arithmetic.
There is a standard triviality that forces us to worry about the theory $T$. If we take a sentence $\phi$ that is true in some (at least one) model $M$, then there is a consistent theory in which $\phi$ is provably true: the theory consisting of all sentences that are true in $M$. Therefore, we see that the property of being "provably true" is really a relationship between a sentence and a theory, rather than an intrinsic property of a sentence itself.
Best Answer
The sentence $1=0$ is false, so it is (hopefully) unprovable. On the other hand it is disprovable (in any halfway reasonable theory for reasoning about the integers) and therefore decidable.
In order to be undecidable, both the sentence and its negation must be unprovable. In other words, your "a sentence which is unprovable, and also its negation is unprovable" is exactly what "undecidable" means.
Note that this is always relative to a particular theory or proof system. Something can't just be "undecidable" in and of itself; but it can be "undecidable in ZF" or "undecidable in PA", for example
(Beware that in the related area of computability theory, "undecidable" has a completely different meaning, and is synonymous with "non-computable". The two meanings of "undecidable" do not coincide, and don't even apply to the same kind of things. In particular, "computable" doesn't apply to single sentences at all.)
Such sentences are necessarily disprovable (because if they were not, they would be undecidable). So if we try to add them as axioms, we get an inconsistent theory.
Yes, if they are taken to be about truth in the "intended model" of whichever language your theory is phrased in, such as the actual integers. Most of us feel intuitively that the integers exist in some Platonic way, independently of any formal systems for reasoning about them, and that all sentences in the language of arithmetic have a definite (though not necessarily known) truth value when applied to the actual integers.