Following Willie Wong, the answer to his question depends on the cardinality of $X$.
In the case when $card(X)=c$ his question is not uniquely solvable
in the theory $(ZF)\&(DC)$ for $X=[0,1]$ and $\cal{F}=P[0,1]$, where
$(ZF)$ denotes the Zermelo-Fraenkel set theory and $(DC)$
denotes the axiom of Dependent Choices.
Indeed, on the one hand, in the consistent theory $ZF \&DC \& AD$, where $AD$ denotes an Axiom of Determinacy, the answer to his question is yes, because Mycielski and Swierczkowski well known result asserts that every subset of the real axis is Lebesgue measurable. Hence such a measure is exactly Lebesgue measure in $[0.1]$.
On the other hand, in the consistent theory $ZF\& DC \& AC \& \omega_1=2^{\omega}$ by Ulam's well known result on the powerset of $\omega_1$ (correspondingly, of $2^{\omega}$) we can not define a probability measure which vanishes on singletons.
Since both theories are consistent extensions of the theory $ZF\& DC$ we deduce that Willie Wong's question is not solvable within the theory $ZF\&DC$ for $X=[0,1]$ and $\cal{F}=P[0,1]$.
Formally, measure (resp. probability) theory requires us to works with a triple $(\Omega, \mathcal{F}, P)$ where $\Omega$ is the space we are working on, $\mathcal{F}$ is a $\sigma-$algebra and $P$ is a (probability) measure which maps elements of $\mathcal{F}$ to numbers (between $0$ and $1$). We call the elements of $\mathcal{F}$ the "$\mathcal{F}$ measurable sets". For any non-trivial $\Omega$, you will have many potential $\sigma-$algebras that you can use in the place of $\mathcal{F}$. As you say, one option is to take $\mathcal{F} = 2^\Omega$ (the power set of $\Omega$) to be our $\sigma-$algebra. The problem with this choice is that every subset of $\Omega$ is in the power set--everything is measurable here. Why is that an issue? Among other things, it's often too big for $P$ to have nice properties. In many (most, honestly) cases, finding nice properties we want $P$ to have is what really drives the probability, not the particulars of $\mathcal{F}$.
Stefan gives the standard non-probabilistic example of this in the comments. The Lebesgue measure, which is the natural notion of volume on the real line, is not compatible with the power set as the $\sigma-$algebra in our triple, so we need to pick a new one. The definition of the Borel $\sigma-$algebra is that it is the smallest $\sigma-$algebra containing the open intervals (which had better be measurable if we are going to define volume). Since this $\sigma-$algebra is compatible with the intuitive notion of volume, it is therefore the smallest $\sigma-$algebra we can choose with the property that $\mu\{(a,b)\} = b-a$ for all open intervals $(a,b)$. Why not stop here? Not all subsets of Borel sets of measure $0$ are measurable and it is often nice for the sake of theory to not have to worry about those sets. The Lebesgue $\sigma-$algebra is what you get if you insist that all subsets of sets of measure zero are measurable. In this case, as in many cases, because this $\sigma-$algebra is so natural we often drop the formalism and just say that a set is "measurable" or "not measurable" on the real line, when what we really mean is that it is measurable with respect to the Lebesgue $\sigma-$algebra or not measurable with respect to the Lebesgue $\sigma-$algebra. I believe that the last issue is the source of your confusion. Whatever you were reading dropped that they were referring to the Lebesgue $\sigma-$algebra.
It is not too difficult and not too trivial to construct a set which is Lebesgue measurable but not Borel measurable. In general, most sets you can write down will end up being Borel. By contrast, constructing sets which are not Lebesgue measurable requires using something like the axiom of choice. Analysts are fond of saying that if you can write it down explicitly, it is Lebesgue measurable.
Let me make one quick comment about why probabilists like to use the Borel $\sigma-$algebra rather than the Lebesgue $\sigma-$algebra. For an analyst, the definition of a function being measurable is that the inverse image of open sets is measurable. Since probabilists don't require our spaces to have topologies, this really doesn't work for us. For a probabilist, the definition of a function being measurable is that the inverse image of a measurable set is measurable. The Borel $\sigma-$algebra has the nice property that if you compose two Borel measurable functions, you get another Borel measurable function in either definition. This property fails badly for Lebesgue measurable functions with the analysts' definition of measurable.
Best Answer
The comment in the last sentence in exercise 1.4.25 from Terry Tao's Measure Theory is about measures, not probabilities.
The OP asks: "Anyone knows a simple example of a nondiscrete measure on the power set of an uncountable set?"
Here is a simple example. Let $X$ be any uncountable set and let $2^X$ be the power set of $X$. For every $E \in 2^X$, define $\mu(E) = 0$, if $E$ is countable and $\mu(E)=+\infty$, if $E$ is uncountable.
It is easy to prove that $\mu$ is a measure defined on $2^X$. It is also easy to see that for any singleton $\{a\} \in 2^X$, $\mu(\{a\})=0$. So, $\mu$ is a non-discrete measure ($\mu$ can not be represented in the form $\mu = \sum_{x \in X} c_x \delta_x$ for some ${c_x \in [0,+\infty]}$).