I am trying to understand the procedure of carrying out a Bayesian Repeated Measures ANOVA. In a conventional repeated measures ANOVA, I calculate the effect of a certain parameter (e.g., study condition) on a measured variable (e.g., test score). Then, I follow up my test with a post hoc test if the p value is lower than a certain value. This post hoc test will tell me the effect of each individual study condition. Therefore, I do not only know that the study condition had an effect on the measured results – I am also able to infer which condition had the most effect.
For Bayesian Repeated Measures ANOVA, it is not clear to me how to follow up the initial test. I can do a 'model comparison', which gives me a Bayesian Factor to indicate the effect of that specific model. Lets say study condition is found to have a high BF – now I am of course interested which study condition performed better / worse than the others. How do I approach this question?
Additional information about the study: This concerns a within-subjects, full repeated measures design. The study consisted of a total of three conditions – with participants completing the different conditions in different orders (i.e., balanced latin square).
(I used JASP software for my analysis, whereas the regular repeated measures ANOVA offers the possibility for post hoc testing, this is not available in the Bayesian repeated measures ANOVA.)
Best Answer
First, why do you want to perform a Bayesian analysis? If you want to know which of your conditions is "significantly different" from the others, the repeated-measures ANOVA with planned contrasts gives you the answer. The strength of the Bayesian approach is not in this kind of null-hypothesis significance testing (NHST), but rather in parameter estimation. The basic idea is that under the Bayesian approach, you are never boiling down your model to binary decisions about accepting/rejecting models, nor assuming "point estimates" for your parameters, allowing you to evaluate the performance of your model in more detail. For more information, see Kruschke's excellent book or this paper, in particular the section entitled "Bayesian estimation generally". Previous questions on Cross Validated are well worth a read (here and here).
Now onto your specific problem. Let $n$ represent the number of subjects, $m$ represent the number of conditions (in this case, $m=3$), and $Y$ represent a stacked vector of your data, organized like this:
$$ Y = \begin{pmatrix} y_{1} \\ y_{2} \\ \vdots \\ y_{n\cdot{}m} \end{pmatrix}, $$
where the $y_1$ to $y_m$ are the values from the first subject, $y_{m+1}$ to $y_{2m}$ are the values from the second subject, and so on. We assume that the data are normally distributed,
$$Y\sim\mathcal{N}\left(\vec{\mu},\sigma^2\right),$$
where $\vec{\mu}$ is a stacked vector of means and $\sigma^2$ is the variance. The mean vector is given by
$$\vec{\mu}=\textbf{X}\vec{\beta},$$
where $\textbf{X}$ is a design matrix and $\vec{\beta}$ is a row vector of regression coefficients. The design matrix is a bit complicated due to the repeated-measures design. Each row represents a data point in $Y$. The first $n$ columns in the matrix represent subject-specific "intercepts"; for example, in the first column, the first three values are 1 and all the rest are 0. The remaining columns represent conditions.The first condition is absorbed by the intercept, so the $(n+1)$th column has a 1 for every data point belonging to condition 2, the $(n+2)$th column has a 1 for every data point belonging to condition 3, and so on.
The row vector $\vec{\beta}$ looks like
$$\vec{\beta} = \begin{pmatrix} \beta_1 & \beta_2 & \cdots & \beta_{n\cdot{}m-1} \end{pmatrix},$$
where each entry is a normally distributed random variable with a suitable prior. Finally, we need to assign a prior to $\sigma$, such as a half-Cauchy.
And that's it! You can construct this model in BUGS, JAGS, Stan, or PyMC, and sample the joint posterior using MCMC. To determine whether there was a difference between, for example, condition 1 and condition2, look at the marginal posterior distribution of $\beta_{n+1}$. This represents the group-level average difference between these conditions. If the 95% credible interval of this distribution excludes 0, you can conclude that there is probably a meaningful difference between condition 1 and condition 2.
PS: I really, really hate the term "Bayesian ANOVA". I know it crops up all the time and there is nothing wrong with it technically, but I find that for the uninitiated, it implies a too much equivalence between the frequentist and Bayesian approaches.