I understand the effect of a categorical random effect on a mixed effects model in that it performs a partial pooling of the observations by level in the random effect, effectively assuming that the observations aren't independent themselves but only their partial pools are.
Also to my understanding, in such a model observations sharing the same random effect level but differing in their fixed effect level will outweigh observations differing in both their random effect and fixed effect levels.
What is the effect of a continuous random factor then?
Given that a model without the random effect showed that the fixed effect had an effect size X. Should I expect that if the observations in the different levels of the fixed effect came from far ends of the random effect continuum the effect size will become smaller in a model which included the random factor, while if observations in different fixed factor levels had similar random effect values then the effect size would increase?
Best Answer
I had to think hard about what you were asking. At first I thought along the lines of @user11852, that you were wanting every observation to have its own unique random effect. That would make the model hopelessly unidentified, as there would be no conceivable way to distinguish random effect variation from the model error.
But I believe that in the scope of your intended question, all random effects are actually continuous, and probably normally distributed. However, your allusion to "categorical" is not off the wall, because the design matrix for a random intercept (typically called Z) would look like a design matrix for a categorical variable.
Let's add a bit of concreteness and say that the linear predictor is $$(\bar{\alpha} + \alpha_i) + (\bar{\beta} + \beta_i) x_{ij},$$ where $\bar{\alpha}$ and $\bar{\beta}$ are the fixed effects and $\alpha_i$ and $\beta_i$ are the $i$-specific random effects. I think that by "continuous," you mean a random effect like $\beta_i$ rather than $\alpha_i$. Note that both of these are still constant within a subject $i$.
Now let's think of your proposed situation:
If we consider $\bar{\beta}$ to be the fixed effect, then it couldn't have different levels, but $x_{ij}$ could. Let's assume that for small values of $x_{ij}$, the slope is smaller; $\beta_i$ is negative for subjects $i$ with mostly small values of $x_{ij}$. Now by construction, the extremes of the $x_{ij}$ correspond to the extremes in $\beta_i$.
That leaves us with what happens with vs without the random effect. My thoughts are, if there were only a few extreme cases of the situation above, adding a random effect would tend to pull the estimate of $\beta$ upwards. But I'm not totally sure. In traditional linear mixed modeling, the estimates of the fixed effects are really just weighted least squares estimates. While those weights are directly related to the random effects distribution, their impact will diminish as your sample size increases. In a realistic setting with even moderate sample sizes, I wouldn't expect anything too extreme to happen to your fixed effect estimates when you add in a random effect.