applicationssoft-questionstatistics
Both Benford's Law (if you take a list of values, the distribution of the most significant digit is rougly proportional to the logarithm of the digit) and Zipf's Law (given a corpus of natural language utterances, the frequency of any word is roughly inversely proportional to its rank in the frequency table) are not theorems in a mathematical sense, but they work quite good in the real life.
Does anyone have an idea why this happens?
(see also this question)
Some of the data generated by Math Stack Exchange itself (and, presumably, by similar sites) ought to approximately follow Benford's law. These would include the distribution of first digits of
This is because Benford's law applies to exponentially growing quantities, and the total number of all of these quantities ought to be growing exponentially. It's only approximate because of artifacts in the way that some of these quantities are determined and because you need several orders of magnitude in order to see Benford's law really kick in.
Anyway, I decided to test this for the first three. The first digits weren't actually that hard to compile because the site allows you to sort the first three from highest to lowest. (Unfortunately, it does not allow you to sort the fourth that way, and I don't feel like wading through 2200+ questions to collect the data.) Here are the results I got on the first three.
Tag Use
(Data collected October 25, 2010.)
Not a bad fit, especially when you consider that there are only three orders of magnitude represented in this measure. There are a disproportionately large number of tags that were created but only used once, which explains the larger frequency of 1 as a first digit.
Votes Per Question
(Data collected October 26, 2010.)
Also not a bad fit, especially since there are currently only two orders of magnitude represented in this measure. Also, I threw out the questions with 0 and negative numbers of votes.
User Reputations
(Data collected October 25 and 26, 2010.)
This is the worst fit of the three, as the frequency of 1 as a first digit is so much larger than the others. However, there are a very large number of users who have never posted a question or an answer and so have a rep of 1. And, for reasons unknown to me, there are also a large number of users who have a reputation of 101, despite never having asked or answered a question.
If you remove 1 as a possible first digit and then rescale the Benford law probabilities to consider only 2 through 9 as possible first digits then the picture looks like the following, which is a much better Benford fit.
Admittedly, none of these data sets would pass Pearson's chi-square test for goodness-of-fit with respect to the Benford probabilities at a reasonable level of significance. However, given some of the artifacts in the data and the fact that there are relatively few orders of magnitude represented, the fit with Benford's law is really not that bad.
The linked paper is titled unfortunately, at least as regards the current conception of the word "random." The whole point of Benford's law is precisely that it doesn't hold when integers are drawn uniformly from a range that, like yours, ends at a power of $10$: a well-designed pseudorandom number generator should give numbers with asymptotically exactly a $\frac{1}{10}$ chance of each leading digit $0,1,...,9$ in decimal notation.
Benford's law applies not to properly random sources of data, but to data produced by certain real-life random processes. One simple characterization of data that obey Benford's law is as those produced by processes that exhibit more-or-less exponential growth, such as populations. Such processes produce uniformly distributed logarithms of data points, but this uniform distribution gets splayed out into the characteristic Benford's law upon exponentiation.
Best Answer
As a rough/somewhat-intuitive explanation of why Benford's Law makes sense, consider it with respect to amounts of money. The amount of time(/effort/work) needed to get from \$1000 to \$2000 (100% increase) is a lot greater than the amount of time needed to get from \$8000 to \$9000 (12.5% increase)--increasing money is usually done in proportion to the money one has. Thought about in the other direction, it should take a fixed amount of time to, say, double one's money, so going from \$1000 to \$2000 takes as long as from \$2000 to \$4000 and \$4000 to \$8000, so the leading digit spends more time at lower values than at higher values. Because the value growth is exponential, the time spent at each leading digit is roughly logarithmic.