Presence (or better sense of presence, i.e. the extent to which one feels that a mediated experience is not actually mediated) is a notoriously difficult variable to pin down and the question of how to measure it is hotly debated in its respective community. One particular debate concerns the the validity of standard self-report questionnaires used to measure a subject's sense of presence.
In a between-subjects experiment, people performed a search task in either a real or a virtual office. They then rated their sense of presence on a set of standard questionnaires. Participants in the virtual condition self-reported a similar sense of presence than participants in the real world condition.
While the authors only conclude that presence questionnaires should pass a "reality test" in order to be useful, I personally think that the results might have been different in a within-subjects experiment. Presence is such a subjective and abstract concept to the layman that I think it unlikely that subject A's feeling of presence under condition X can be meaningfully compared to subject B's feeling of presence under condition Y, but that condition X gives a subject a frame of reference on which to evaluate condition Y, if the subject is exposed to both (Note, IMO this would entail that presence could only be regarded a measure for relative differences between conditions rather than an absolute scale). This is somewhat substantiated by the fact that answers on presence questionnaires have a notoriously high variance.
I have already designed an experiment (which has four different conditions) as a within-subjects experiment using 48 subjects and balancing the order of conditions across subjects.
Then I got to wonder whether I could test my assumption with my data. So I am thinking about analysing my data as within-subjects data (i.e., using a repeated-measures ANOVA) but also taking the very first condition each subject is exposed to and simply treat that subset of the data as if it came from a between-subjects experiment.
However, I am wondering whether this is a valid statistical approach and what statistical test would be appropriate, rather than simply stating, for example, that in the within analysis condition X was different from condition Y while in the between it was not.
For example, would it be valid to treat mean for condition X using within data and mean for condition X using between data (for all four conditions) as factors for an ANOVA to see if the mean for a condition significantly differs for a within and a between subjects experiment?
NB: I could potentially add more subjects which only get exposed to one condition in order to equalise the number of data points available for the within and between subjects analysis but I would prefer to do the analysis on the original subjects only, as (a) the experiment is quite lengthy and cumbersome, and (b) I was actually hoping that keeping the same pool of subjects for both analyses would give me a stronger indication of whether my hypothesis holds, as it eliminates any variability due to some subjects appearing in one set of data but not the other.
Best Answer
The analysis you want to pursue is not uncommon (unfortunately I don't have a reference right now), but you have to split it. You have to run 2 ANOVAs and hope that the analyses agree.
If the seond analysis is not significant but shows the same pattern, this may be, as there is less power.