I would recommend thinking of your functions as composite functions:
$g(x) = \lceil x^2 \rceil = g_2(g_1(x))$, where $g_1(x) = x^2$ and $g_2(x) = \lceil x \rceil$.
To find the preimage, we work backwards. First, we find the preimage $g_2^{-1}(A) = (3,8]$.
The reason 3 isn't in that preimage, is because the ceiling of 3 is 3, which isn't in $A$. Similarly, the reason that the preimage only goes up to 8, is because for any number larger than 8, say 8.01, the ceiling is 9, which isn't in $A$.
Now that we have the preimage $g_2^{-1}(A) = (3,8]$, we can find the preimage preimage $g_1^{-1}\big( (3,8] \big)$, which will be the answer to the overall question.
To find that preimage, we can break up the domain into two regions (and I'm assuming that the domain of the function isn't given, and is assumed to be the largest possible subset of $\mathbb{R}$ on which the expression is well defined, which in this case is all of $\mathbb{R}$).
On the region $x \in [0,\infty)$, since $g_1$ is continuous and strictly increasing and $(3,8]$ is an interval, the preimage is:
$g_1^{-1}\big( (3,8] \big) = ( g_1^{-1}(3), g_1^{-1}(8)] = ( \sqrt{3}, \sqrt{8}]$.
Note that in the above equations, the first $g_1^{-1}$ stands for "preimage", but in the center the two $g_1^{-1}$ stand for the inverse of the function, which is the positive square root, since we're in the region $x \in [0,\infty)$.
On the region $x \in (-\infty, 0)$, since $g_1$ is continuous and strictly decreasing and $(3,8]$ is an interval, the preimage is:
$g_1^{-1}\big( (3,8] \big) = [ g_1^{-1}(8), g_1^{-1}(3)) = [ -\sqrt{8}, -\sqrt{3})$.
The overall preimage is the union of these, which is the answer that you already know.
Hopefully this clears up the process.
Applying the same technique to the example that your teacher did quickly:
$h_2(h_1(x)) = \lfloor 3x-1 \rfloor$, with $h_1(x) = 3x-1$ and $h_2(x) = \lfloor x \rfloor$.
$h_2^{-1}\big( (4,10) \big) = [5,10)$.
That's the trickiest part, so let me explain why. Since we're dealing with the floor function this time, if $x \in (4,5)$ then it will be mapped to 4, which isn't in $(4,10)$. So our preimage starts at 5. On the other hand, if $x \in (9,10)$, then it will be mapped to 9, which is in $(4,10)$, so our preimage can go all the way up to, but not including, 10.
The next step is easier than in your problem, since $h_1$ is continuous and strictly increasing on all of $\mathbb{R}$:
$h_1^{-1}\big( [5, 10) \big) = \big[ h_1^{-1}(5), h_1^{-1}(10) \big) = \big[ 2, 11/3 \big)$
If you ever have trouble reasoning through the preimage as I've done above, a slower but less error prone way to find the preimage is to graph the function, then mark on the y axis the set that you want to find the preimage for, and use the graph to see what set on the x axis is the preimage of that.
Best Answer
One domain is $\{\rho_1(x,y)<1\}$, the other is $\{\rho_2(x,y) <1\}$, where $\rho_1(x,y) =\sqrt{x^2+y^2}$, and $\rho_2(x,y) = \max(|x|,|y|)$. Find a map $F\colon (x,y) \mapsto (x',y')$ so that $\rho_1(x,y) = \rho_2(x',y')$. It should be linear on each line through the origin. One can take: $$F(x,y) = \frac{\rho_1(x,y)}{\rho_2(x,y)} \cdot (x,y)$$ that is \begin{eqnarray} F(x,y) = \frac{\sqrt{x^2+y^2}}{\max(|x|,|y|)} \cdot (x,y) \end{eqnarray} with inverse \begin{eqnarray} F^{-1}(x,y) = \frac{\max(|x|,|y|)}{\sqrt{x^2+y^2}} \cdot (x,y) \end{eqnarray}
$F$ provides a homeomorphism from the disk to the square.