This is many years later, but it turns out there is such a transform, which is a simple variant of the usual bilateral Mellin transform.
First, we must say what a bilateral Mellin transform is. One very common method of doing this, which will not quite work for our situation, is to take the Mellin transform of the positive and negative halves of the signal separately. The pair of the results $\left(\mathscr{M}_- f, \mathscr{M}_+ f\right)$ is thus treated as being the bilateral Mellin transform of the original signal. (Note that some care needs to be taken regarding the value of $f$ at 0.)
However, it is not hard to see that this does not yield the property that bilateral Mellin convolution does not correspond the independent product of the positive and negative transforms in this domain. It is almost that, but rather something slightly more complicated (the details are below).
It turns out these problems can be solved if, rather than taking "positive" and "negative" Mellin transforms, we take "even" and "odd" Mellin transforms. That is, we split our function into even and odd addends, such that $f_e(t) + f_o(t) = f(t)$, and then the one-sided Mellin transforms of the even and odd parts. This happens to give the correct result, which is that bilinear Mellin convolution in the time domain translates to pointwise multiplication in this domain.
The proof is pretty simple, and mostly involves a bunch of elementary but tedious integral manipulations for which the primary challenge is not to drop any signs, factors of 2, etc, the details of which are below. Note also that this is basically the same transform as the positive/negative approach, as the odd and even Mellin transforms will just be simple linear combinations of the positive and negative ones, just in a different basis that we have chosen to have this pointwise multiplication property.
To see this, first suppose we're Mellin-convolving $f(t)$ and $g(t)$, and writing the result as $(f \ast_\widehat{M} g)$. Then we can split $f(t) = f_{-}(t) + f_{+}(t)$, where we have
$$
f_{-}(t) = \left\{ f(t) \text{ if } t < 0, 0 \text{ otherwise} \right\}\\
f_{+}(t) = \left\{ f(t) \text{ if } t\geq 0, 0 \text{ otherwise} \right\}\\
$$
And similarly with $g$, so that we get
$$
(f \ast_\widehat{M} g) = (f_- + f_+) \ast_\widehat{M} (g_- + g_+)
$$
Then, because Mellin convolution distributes on addition, we can treat the result as a sum of four separate sub-convolutions:
$$
(f_{-} \ast_\widehat{M} g_{-}) + \\
(f_{-} \ast_\widehat{M} g_{+}) + \\
(f_{+} \ast_\widehat{M} g_{-}) + \\
(f_{+} \ast_\widehat{M} g_{+})
$$
Since the Mellin transform is linear, the Mellin transform of the sum of all the above will be the sum of the Mellin transforms of each of the four sub-terms above (and similarly for any of these positive/negative variations). However, we can easily see that the pairs with the same sign will convolve to a function which is nonzero for only positive values, and those with different sign will be nonzero for only negative values. Thus, the positive Mellin transform of any pair with different sign will be zero, and the negative Mellin transform of any pair with the same sign will be zero. Thus, we get have
$$
\mathscr{M}_{+}(f \ast_\widehat{M} g) = (\mathscr{M}f_+) (\mathscr{M}g_+) + (\mathscr{M}f_-) (\mathscr{M}g_-) \\
\mathscr{M}_{-}(f \ast_\widehat{M} g) = (\mathscr{M}f_+) (\mathscr{M}g_-) + (\mathscr{M}f_-) (\mathscr{M}g_+)
$$
where we will use the following convention for the positive and negative Mellin transforms
$$
\mathscr{M_+}f(t) = \mathscr{M}f_+(t) = \int_0^\infty f(t) t^{s-1} dt \\
\mathscr{M_-}f(t) = \mathscr{M}f_-(-t) = \int_0^\infty f(-t) t^{s-1} dt = \int_{-\infty}^0 f(t) (-t)^{s-1} dt \\
$$
This is not quite what we wanted, which is some "domain" for which the bilateral convolution is a simple pointwise multiplication.
Making the slight modification of using even and odd parts of our original functions, rather than positive and negative parts, turns out to give us what we want. We will define the even and odd part of our function in the usual manner as
$$
f_e(t) = \frac{f(t) + f(-t)}{2} \\
f_o(t) = \frac{f(t) - f(-t)}{2}
$$
We again can split the original bilateral convolution into a sum of the following four sub-convolutions:
$$
(f_{e} \ast_\widehat{M} g_{e}) + \\
(f_{e} \ast_\widehat{M} g_{o}) + \\
(f_{o} \ast_\widehat{M} g_{e}) + \\
(f_{o} \ast_\widehat{M} g_{o})
$$
Now, the critical step: it is very easy to see that the bilateral Mellin convolution of any pair of even functions will be even, of any pair of odd functions will be odd, and that if you bilaterally convolve an even and an odd function the result will be zero, although we won't prove it explicitly here. As a result, when we take the Mellin transform of these four terms and add the results together, we get
$$
\mathscr{M}_{e}(f \ast_\widehat{M} g) = (\mathscr{M}_{e}f) (\mathscr{M}_{e}g) \\
\mathscr{M}_{o}(f \ast_\widehat{M} g) = (\mathscr{M}_{o}f) (\mathscr{M}_{o}g)
$$
where we have
$$
\mathscr{M_e}f = 2 \mathscr{M}f_e = \int_0^\infty f_e(t) t^{s-1} dt \\
\mathscr{M_o}f = 2 \mathscr{M}f_o = \int_0^\infty f_o(t) t^{s-1} dt
$$
where the factor of two is important to make sure things are scaled correctly with respect to the resulting pointwise multiplication, as we are now only taking the one-sided transform of a two-sided function.
Thus, we do get a simple product in this situation. Thus, if we treat the resulting pair of even and odd transforms as "the" bilateral Mellin transform, then pointwise multiplication in that domain is bilateral Mellin convolution in the time domain. This is probably the simplest real answer to the question.
However, if we really want to be cute, we can go one step further and get a "single transform" out of all of this, rather than having this pair of transforms, although it's kind of just for fun.
The basic idea is, rather than treating the even and odd Mellin transforms as this pair of individual transforms, we can treat the bilateral Mellin transform as a single function of one variable that takes values in $\Bbb C^2$, treated as a ring with pointwise addition and multiplication. That is, we can write
$$
\widehat{\,\mathscr{M}} f = \left(\mathscr{M}_e f, \mathscr{M}_o f \right)
$$
This becomes slightly more interesting if we note that the ring $\Bbb C^2$ is isomorphic to the $\Bbb T$, the commutative ring of "tessarines" or "bicomplex numbers". This is a 4D real algebra (or a 2D complex algebra) in which all elements are of the form
$$
a + bi + cj + k
$$
with
$$
i^2 = -1, j^2 = 1, k^2 = -1, ijk=-1
$$
The $j^2 = 1$ makes these of a somewhat different flavor from quaternions, as does the commutativity, and the fact that there are zero divisors, which will not bother us for this application.
The elements $e = \frac{1+j}{2}$ and $o = \frac{1-j}{2}$ are idempotent, and thus in our isomorphism we will map $e = (1,0)$ and $o = (0,1)$ into them respectively. Doing some algebra we can thus represent our original bilateral Mellin transform as a tessarine-valued function:
$$
\widehat{\,\mathscr{M}} f = \left(\mathscr{M}_e f, \mathscr{M}_o f\right) \\
= \mathscr{M}_e f \left(\frac{1+j}{2}\right) + \mathscr{M}_o f \left(\frac{1-j}{2}\right) \\
= \left(\mathscr{M}_e f + \mathscr{M}_o f\right) \frac{1}{2} + \left(\mathscr{M}_e f - \mathscr{M}_o f\right) \frac{j}{2}
$$
However, we have the interesting identities
$$
\mathscr{M}_e f(t) = 2 \mathscr{M} f_e(t) \\
= 2 \mathscr{M} \left(f(t) + f(-t)\right) \frac{1}{2} \\
= \mathscr{M_+} f + \mathscr{M_-} f
$$
and
$$
\mathscr{M}_o f(t) = 2 \mathscr{M} f_o(t) \\
= 2 \mathscr{M} \left(f(t) - f(-t)\right) \frac{1}{2} \\
= \mathscr{M_+} f - \mathscr{M_-} f
$$
Plugging back into the above, at last, we get
$$
\widehat{\,\mathscr{M}} f= \left(\mathscr{M}_e f + \mathscr{M}_o f\right) \frac{1}{2} + \left(\mathscr{M}_e f - \mathscr{M}_o f\right) \frac{j}{2} \\
= \left(\mathscr{M_+}\right) f + \left(\mathscr{M_-} f\right) j
$$
which does have the property that
$$
\widehat{\,\mathscr{M}} (f \ast_\widehat{M} g) = (\widehat{\,\mathscr{M}} f) (\widehat{\,\mathscr{M}} g)
$$
So that in a way we do end up back with our original idea of the positive and negative Mellin transforms, just embedded into the tessarines in such a way that the pointwise tessarine multiplication gives the correct behavior for bilateral convolution.
There is a natural extension of this idea to the distributional setting, where we have, for instance
$$
\widehat{\,\mathscr{M}} \left\{\delta(t-1)\right\} = 1 \\
\widehat{\,\mathscr{M}} \left\{\delta(t+1)\right\} = j \\
\widehat{\,\mathscr{M}} \left\{\frac{\delta(t-1) + \delta(t+1)}{2}\right\} = \frac{1+j}{2} = e = (1,0) \\
\widehat{\,\mathscr{M}} \left\{\frac{\delta(t-1) - \delta(t+1)}{2}\right\} = \frac{1-j}{2} = o = (0,1) \\
$$
and so on, where $(1,0)$ refers to our earlier representation of the algebra as a direct product of $\Bbb C$'s.
Lastly, it turns out if we want the entire thing in integral form, we can do some more fiddling about to ultimately arrive at
$$
\widehat{\,\mathscr{M}}f = \left(\mathscr{M_+}\right) f + \left(\mathscr{M_-} f\right) j \\
= \left(\int_0^\infty f(t) t^{s-1} dt\right) + \left(\int_{-\infty}^0 f(t) (-t)^{s-1} dt\right) j \\
= \left(\int_0^\infty f(t) |t|^{s-1} dt\right) + \left(\int_{-\infty}^0 f(t) |t|^{s-1} dt\right) j \\
= \int_{-\infty}^\infty f(t) |t|^{s-1} \mathbf{j}(t) dt
$$
where $\mathbf{j}(t) = \left\{ j \text{ if } t<0, 1 \text{ otherwise} \right\}$.
Best Answer
Fix an $x \in (0, 1)$. If $h$ has a singularity at $1$ and we define the convolution as an improper integral, we obtain $$f*h = \lim_{y \downarrow x} \int_y^1 f(t) h {\left( \frac x t \right)} \frac {dt} t = \lim_{y \downarrow x} G(y), \\ h*f = \lim_{y \uparrow 1} \int_x^y h(t) f {\left( \frac x t \right)} \frac {dt} t = \lim_{y \uparrow 1} \int_{x/y}^1 f(t) h {\left( \frac x t \right)} \frac {dt} t = \lim_{y \uparrow 1} G {\left( \frac x y \right)}.$$ When $y$ tends to $1$ from below, $x/y$ tends to $x$ from above and $x/y \neq x$ (since $x \neq 0$). Under these conditions, the limit composition rule holds: either $$\lim_{y \uparrow 1} G {\left( \frac x y \right)} = \lim_{y \downarrow x} G(y)$$ or both limits do not exist.