From your explanation for AND I take it that you're working with the usual Church booleans:
$\top = [\lambda x,y. x]$,
$\bot = [\lambda x,y. y]$.
You then define AND using $\bot$ as follows:
$\land = [\lambda x,y.(x~y~\bot)]$,
which, as your correctly assessed, has the desired property of selecting the minimum of x and y.
When x is $\bot$ we "short-circuit" to $\bot$: $(\land~\bot~\circ) \Rightarrow \bot$, since $(\land~\bot~\circ)$ $\equiv$ $[\lambda x,y(x~y~\bot)](\bot)(\circ)$
$\rightarrow_{\beta} [\lambda y(\bot~y~\bot)](\circ)$
$\rightarrow_{\beta} (\bot~(\circ)~\bot) = \bot$.
When x is $\top$, then we reduce to $y$: $(\land~\top~\circ) \Rightarrow (\circ)$, since $(\land~\top~\circ) \equiv$ $[\lambda x,y(x~y~\bot)](\top)(\circ)$
$\rightarrow_{\beta}$ $[\lambda y(\top~y~\bot)](\circ)$
$\rightarrow_{\beta}$ $(\top~(\circ)~\bot) = (\circ)$.
To get exclusive or let's first define $\lnot$, so that we have a functionally complete set of operators:
$\lnot = [\lambda b,x,y. (b~y~x)]$.
Recall what our truth-values $\top,\bot$ do: they take terms $x,y$ and the first returns $x$, the second $y$. Intuitively, $\lnot$ is plausible for it swaps $x$ and $y$, so that $\top$ returns $y$ and $\bot$ returns $x$. Let's check:
When x is $\top$ we want to get $\bot$. Since $(\lnot \top) \equiv [\lambda b,x,y. (b~y~x)]~([\lambda x,y. x])$
$\rightarrow_{\alpha}[\lambda b,u,v. (b~v~u)]~([\lambda x,y. x])$
$\rightarrow_{\beta} [\lambda u,v. (([\lambda x,y. x])~v~u)]$
$\rightarrow_{\beta} [\lambda u,v. v)]$
$\rightarrow_{\alpha} [\lambda x,y. y)] = \bot$.
When x is $\bot$ we want to get $\top$. Since $(\lnot \bot) \equiv [\lambda b,x,y. (b~y~x)]~([\lambda x,y. y])$
$\rightarrow_{\alpha}[\lambda b,u,v. (b~v~u)]~([\lambda x,y. y])$
$\rightarrow_{\beta} [\lambda u,v. (([\lambda x,y. y])~v~u)]$
$\rightarrow_{\beta} [\lambda u,v. u)]$
$\rightarrow_{\alpha} [\lambda x,y. x)] = \top$.
Using $\land$ and $\lnot$, we can define inclusive or ($\lor$) in the usual way:
$\lor = [\lambda x,y.\lnot(\land~(\lnot x)~(\lnot y))]$.
I'll leave it to you as an exercise to verify that it acts as a disjunction operator. Now exclusive or is:
$\oplus = [\lambda x,y.\land~(\lor~x~y)~(\lnot~(\land~x~y))]$,
which, as is expected, will take terms $x,y$ and return $\top$ if $x\not=y$ and $\bot$ otherwise. I'll let you verify.
Best Answer
It's just conventions. Underlyingly, everything is unambiguously bracketed, it's just that we're lazy and sometimes want to omit brackets for easier readability and writability, and we need to establish conventions on which term we mean (the term with the one or with the other application/abstraction structure) when we leave brackets away. These are the precedence rules. If the source is good, it will explicitly specify the precedence rules and state whether terms with not all of the brackets present are to be read as one or as the other term of the two you listed. Application having precedence over abstraction is the more usual convention, but defining it the other way round is just as "correct" as long as it's done consistently. So the interpreter is not "wrong", it's just a bit sloppy for not stating its precedence conventions explicitly, or rather, contradicting the precedence conventions used in the source it references at the bottom of the page.
If you want to evaluate your term $ \lambda f x. f (f x)$ in the interpreter but with the application preceding abstraction reading, you just need to explicitly indicate the intended structure by providing all the necessary brackets $ \lambda f. \lambda x. (f (f x)) $ in order to override the interpreter's default reading as $ \lambda f .(\lambda x. f) (f x)$.