Solved – Compare two samples from same population taken at different times

chi-squared-testhypothesis testingproportion;

I have two samples (both n > 1000) taken from the same population. The first sample is taken in one year, say at t0, and the the other sample is taken a year after, say at t1. There are different people in the different samples, i.e. I do not have two measurements from the same person at t0 and t1. How do I test whether some proportion in the population has changed from t0 to t1?

My initial thought was to do a chi-squared test for difference in proportion, but I am not sure whether this is valid when the two samples are not from two populations. I have also considered doing a one-way within groups ANOVA.

Best Answer

If both samples come from the same population then they would have the same characteristics but since due to sampling there is randomness involved, we could not be able to measure it precisely and we could obtain different sample characteristics. Comparing two samples from the same population would only tell you something about how precise are your measurements.

If you are comparing individuals as measured on time $t$ to individuals as measured on time $t+k$, then you are dealing with two samples, each taken from the different population. Population in statistics is simply a group of cases sharing some property of interest.

Check this question for a discussion of a similar misconception. Check also here, here and here.

Related Solutions

Testing Multinomial Samples for Same Distribution

You correctly performed a $\chi^2$-test of independence, so the only problem is in the formulation of its hypotheses and the interpretation of the test result:

The $\chi^2$-test of independence tests the null hypothesis "The two color distributions are equal" versus the working hypothesis of any difference. The p value is smaller than the prespecified level $\alpha$, so you reject the null hypothesis and claim with about $(1-\alpha)\cdot 100\%$ confidence that the colors are differently distributed between urns.

The term "independence"-test is sometimes a bit confusing but it is more clear if you consider the "raw" data behind the contingency table:

Color   Urn
Blue      1
Blue      2
Green     2
Red       1
Blue      1
...

The null hypothesis that the variable "Urn" is independent of the random variable "Color" is equivalent to the null hypothesis stated above. So it's not about independence of the two color distributions but about independence of color and urn.

Note that a large p value wouldn't mean that the color distributions were equal. This would be much harder to show by "classic" statistical methods.

Hypothesis Testing – How to Check if Proportions in Two Small Samples are the Same?

The reason your sample size is a problem for a chi-square test or a test of proportions that relies on approximation to a Normal distribution is that these tests use asymptotic results. In other words, they are built on the assumption that you have "large enough" sample size.

A test that is similar to the chi-square test, but does not rely on asymptotic assumptions, is Fisher's exact test described here: http://en.wikipedia.org/wiki/Fisher%27s_exact_test

Best Answer

Related Solutions

Testing Multinomial Samples for Same Distribution

Hypothesis Testing – How to Check if Proportions in Two Small Samples are the Same?

Related Question