The first thing you should do is challenge your lecturer to explain these things clearly. If anything whatsoever seems counter-intuitive or backwards, them demand that he/she explains why it is intuitive. Statistics always makes sense if you think about it in the "right" way.
Calculating pivotal quantities is a very tricky business - I completely understand your bewilderment in "where should I start?"
For normal variance parameters, The "pivotal quantity" is the sum of squares divided by the variance parameters:
$$T_{X}=\sum_{i=1}^{N}\Big(\frac{X_{i}-\mu_{X}}{\sigma{X}}\Big)^{2} \sim \chi^{2}(N)$$
And a similar expression for $T_{Y}$. Note that the distribution only depends on $N$, which is known (if $\mu_{X}$ is unknown, replace by $\overline{X}$ and you lose one degree of freedom in the chi-square distribution). Thus $\frac{T_{X}}{T_{Y}}$ is a pivotal quantity, which has a value of:
$$\frac{\sum_{i=1}^{N}\Big(\frac{X_{i}-\mu_{X}}{\sigma{X}}\Big)^{2}}{\sum_{i=1}^{N}\Big(\frac{Y_{i}-\mu_{Y}}{\sigma{Y}}\Big)^{2}}
$$
Note that because it is a pivotal quantity, we can create an exact confidence interval using the pivot as a starting point, and then substituting in our statistic. Now because the degrees of freedom are the same for each chi-square, we do indeed have an F distribution. So you can write:
$$1-\alpha=Pr(L < F < U)=Pr(L < \frac{T_{X}}{T_{Y}} < U)$$
$$1-\alpha=Pr(L < \frac{\sum_{i=1}^{N}\Big(\frac{X_{i}-\mu_{X}}{\sigma{X}}\Big)^{2}}{\sum_{i=1}^{N}\Big(\frac{Y_{i}-\mu_{Y}}{\sigma{Y}}\Big)^{2}} < U)$$
$$1-\alpha=Pr(L < \frac{\sigma_{Y}^{2}}{\sigma_{X}^{2}}\frac{\sum_{i=1}^{N}(X_{i}-\mu_{X})^{2}}{\sum_{i=1}^{N}(Y_{i}-\mu_{Y})^{2}} < U)$$
Writing the observed ratio of the sum of squares as $R$ we get:
$$1-\alpha=Pr(L < \frac{\sigma_{Y}^{2}}{\sigma_{X}^{2}} R < U)$$
$$1-\alpha=Pr(\frac{L}{R} < \frac{\sigma_{Y}^{2}}{\sigma_{X}^{2}} < \frac{U}{R})$$
As for how this solution comes about, I have absolutely no idea. What "principles" were followed (apart from being good at re-arranging statistical expression)?
One thing that I can think of is that you need to find some way to "standardise" your sampling distribution. So for example, normals you subtract mean and divide by standard deviation. For gamma you multiply by the scale parameter. I don't know many pivotal quantities that exist outside of the normal and gamma families.
I think this is one reason why ordinary "sampling statistics" is an art more than a sciecne, because you have to use your intuition about what statistics to try. And then you have to try and figure out if you can standardise your data.
I am almost certain your lecturer will bring up the subject of confidence intervals - be sure to ask him/her what you should do when you only have one sample, or when you have 2 or more nuisance parameters. :)
What you want is the radius $r$ such that the cdf $P(\sqrt{X^2+Y^2}\leq r)=\alpha$ for some alpha. It is clear that $P(\sqrt{X^2+Y^2} \leq r)=P(X^2+Y^2 \leq r^2)$, and it is easier to work with the sums of squared variables.
We can think of $X^2$ and $Y^2$ as given by $X^2=Z_1\sigma_X^2$ and $Y^2=Z_2\sigma_Y^2$ where $Z_1\sim \chi^2_1$ and $Z_2\sim \chi^2_1$ are chi-squared variables with 1 degree of freedom. Given that the chi-squared is a special case of the Gamma distribution, $\chi^2_n = \mathrm{Gamma}(n/2, \frac{1}{2})$ (using the scale-rate parametrization), and using the scaling property of the gamma distribution, we have that $X^2 \sim \mathrm{Gamma}(1/2,\frac{1}{2\sigma^2_X})$ and $Y^2 \sim \mathrm{Gamma}(1/2,\frac{1}{2\sigma^2_Y})$. Thus, we need to find the distribution for the sum of two independent Gamma distributions.
This post gives an expression for the pdf of the sum of two Gamma variables, which you could use to compute the radius suitable for your alpha. Simplifying the expression, for our specific Gammas, the PDF for the sum of squares $S=X^2 + Y^2$:
$P(S) = \left\{\begin{array}{cc}\hfill \frac{e^{-S/(2\sigma_X^2)}}{2\sigma_X\sigma_Y} {}_1F_1\left[\frac{1}{2}, 1, \frac{1}{2} \left(\frac{1}{\sigma_X^2}-\frac{1}{\sigma_Y^2} \right) S \right] \hfill & ,\hfill S >0\hfill \\ {}\hfill 0 & , \hfill S \le 0 \end{array}\right.$
Ideally you'd want the cumulative probability function for your purpose, but I couldn't find such formula. We can obtain it by numerically integrating the PDF though:
import numpy as np
from scipy.special import hyp1f1
import matplotlib.pyplot as plt
# Define pdf function
def radius_squared_pdf(sigmax, sigmay, S):
b1 = 1/(2*(sigmax**2))
b2 = 1/(2*(sigmay**2))
hyp1f1Res = hyp1f1(0.5, 1, (b1-b2)*S)
pdf = (b1**(0.5)) * (b2**(0.5)) * np.exp(-S*b1) * hyp1f1Res
return pdf
# Choose R.V. parameters
sx = 0.5
sy = 1.5
# Set parameters to integrate squared radius PDF
Smax = 20
dS = 0.001
nSteps = int((Smax-dS)/dS)
S = np.linspace(start=dS, stop=Smax, num=nSteps) # squared radius to evaluate
# Evaluate PDF
pdf = radius_squared_pdf(sx, sy, S)
# Integrate PDF
cdf = np.cumsum(pdf*dS)
plt.plot(S, cdf)
plt.ylabel('CDF')
plt.xlabel('Squared radius')
plt.show()
# Find radius corresponding to 95% probability
alpha = 0.95
idx = np.argmin(np.abs(cdf-alpha))
r = np.sqrt(S[idx])
print(f'Radius containing {alpha*100}% of the points: {r}')
Through simulation, we can see below that the match of the analytic pdf to simulations is as we would expect (example, sx = 0.5, sy=1.5, code not shown)):
In sum, what you'd want to do is numerically integrate this formula to find the value of $r^2$ so that $P(S\leq r^2)$ equals your desired $\alpha$. Then the circle you want will have radius $r$. This is shown in the code above.
Best Answer
"Determine to what extent they're different" seems to be more suggesting a problem in estimation rather than testing. I will proceed to discuss tests, but this consideration may suggest you want to do something else.
I'll presume you want independence both within and across groups.
Since $X^2$ and $Y^2$ are Exponential, the likelihood-ratio test for the exponential should yield an equivalent test to the LR test for the Rayleigh. We should then (with a little algebraic manipulation) end up with something equivalent to a test based on the ratio of the means of the squared-observations.
For a one-tailed test an $F$-test would then be obtained pretty directly from the ratio of the likelihoods (that is, a test based on $F=\frac{\overline{x^2}}{\overline{y^2}}$) with $2n_x$ and $2n_y$ df should be equivalent to a likelihood ratio test). For the two tailed case, the "best" test based on the likelihood ratio ($\Lambda$) won't be equal-tailed though for $n$ large it won't matter.
The equal-tail ($α/2$) version of a two-sided test is (in general) not strictly an LRT, (not rejecting the set of smallest values of the likelihood ratio given $α$), since in general one tail will have values of the LR that are smaller than the smallest value in the other tail; you could typically get better power if you actually do the LRT (albeit that may be slightly more complicated to do). See the plot below.
However, a small caveat; from a scientific standpoint you might sometimes prefer to have an equal tail test in spite of this potential loss of power. A critic might say that you're "favouring" one direction of the alternative over the other because it's easier to reject there.
Note that $Λ=\mathcal{L}_0/\mathcal{L}_1$ is the likelihood ratio calculated here under the Rayleigh assumption. You reject when the likelihood under $H_0$ is small relative to the likelihood under $H_1$ – i.e. for small values of the likelihood ratio.
In the case where both $n$'s are large enough that we don't care about giving up a smidge of power for some added convenience, a simple "symmetric" (equal tail) F test would result. While you can do better in terms of power, it is exact and still pretty powerful.
See some discussion of that choice of test here. Note that the df are double the sample sizes. See also Scortchi's answer there for additional enlightening information.
Here's a plot that illustrates the distinction. It is based on 100000 pairs of samples from an exponential(1) (i.e. the squares of your Rayleigh values). The simulation is conducted under $H_0$ here and the specific value of the parameter is immaterial (it cancels). Here the x-sample values are a sample of size 8 and the y-sample values are a sample of size 12. The red values indicate the lower 2.5% and upper 2.5% of an F test (the rejection region for F). The blue dashed horizontal line is the 5% critical value for the likelihood ratio test (estimated here by simply taking the sample quantile from the simulation, though it shouldn't be that hard to actually compute it for this case).
R code used to generate the plot is appended below. I generated a new plot because the old one (by chance it seems) showed an unusually large difference between the tests; typically they're fairly close as with the current plot.
Edit: extra detail where the action is, log-log scale:
The likelihood ratio test rejects some cases in the left tail of the F that the F test does not (i.e. it would place more than 2.5% of the null distribution in the rejection region corresponding the left tail of the ratio of means), and it fails to reject some cases that the F test would reject in the right tail (placing less than 2.5% of the null distribution in the rejection region corresponding the right tail of the ratio of means).
I think the two tests will always correspond when the sample sizes are equal.
However you could also do a permutation test on $Λ$ (i.e. directly on the likelihood ratio), and so still have an exact test even if the Rayleigh assumption wasn't quite right.
A permutation test is straightforward in two-sample test since under $H_0$ observations are directly exchangeable and gives you an exact test in small samples whether or not it's actually Rayleigh but at the same time it should be asymptotically efficient if it is Rayleigh. Doing a permutation version of a likelihood ratio test is not something you see often – to my surprise given it seems to offer a convenient way to have your cake (good power when your distributional model is right) and eat it too (correct significance level even if it's wrong).
If you're not so worried about small-sample exactness, you can use the chi-squared asymptotic null distribution for $-2\log\Lambda$ from likelihood ratio tests, or when working on a computer, you can just fit either a gamma GLM to the square (with appropriate choice of dispersion parameter in the test), or use parametric survival regression and fit a Weibull (again, with one parameter "known").
If you're not at all confident of the Rayleigh (so it might not be "close"), maybe consider a Wilcoxon-Mann-Whitney. If you're nevertheless interested in an interval for the change in the ratio of scales, there's always the possibility of taking logs and getting an interval the shift of the log (which good WMW implementations will give you) and back out interval for the ratio from that.
There's many other things that could be done, but I think these points cover the main things I'd worry about in practice.
Summary
If you're confident it's really Rayleigh, and the sample sizes are very similar or at least not small, use the ratios of the means of the squared data in an F test ($F=\frac{\overline{x^2}}{\overline{y^2}}$) with $2n_x$ and $2n_y$ df.
Note also that there's additional details on this in an answer to this followup question: What is a likelihood ratio test for a specific distribution, and how does it related to hypothesis tests?
Quick bit of R code for the plot:
feetwet (the OP) suggested (via an edit) adding the following Python code. I have accepted the edit, but I HAVE NOT CHECKED THIS CODE, so please keep that in mind.
Same in Python: