Since this question is asked frequently I will try to work out a
solution for generic positive integers $a$ where $a\ge 2$.
Suppose we have $T(0)=0$ and
$$T(1)=T(2)=\ldots =T(a-1)=1$$
and for $n\ge a$
$$T(n) = a T(\lfloor n/a \rfloor) + n \lfloor \log_a n \rfloor.$$
Furthermore let the base $a$ representation of $n$ be
$$n = \sum_{k=0}^{\lfloor \log_a n \rfloor} d_k a^k.$$
Then we can unroll the recurrence to obtain the following exact
formula for $n\ge a$
$$T(n) = a^{\lfloor \log_a n \rfloor}
+ \sum_{j=0}^{\lfloor \log_a n \rfloor -1}
a^j \times (\lfloor \log_a n \rfloor - j) \times
\sum_{k=j}^{\lfloor \log_a n \rfloor} d_k a^{k-j}.$$
Now to get an upper bound consider a string consisting of the digit
$a-1$ to obtain
$$T(n) \le a^{\lfloor \log_a n \rfloor}
+ \sum_{j=0}^{\lfloor \log_a n \rfloor -1}
a^j \times (\lfloor \log_a n \rfloor - j) \times
\sum_{k=j}^{\lfloor \log_a n \rfloor} (a-1) \times a^{k-j}.$$
This simplifies to
$$a^{\lfloor \log_a n \rfloor}
+ \sum_{j=0}^{\lfloor \log_a n \rfloor -1}
a^j \times (\lfloor \log_a n \rfloor - j) \times (a-1)
\sum_{k=0}^{\lfloor \log_a n \rfloor-j} a^k$$
which is
$$a^{\lfloor \log_a n \rfloor}
+ \sum_{j=0}^{\lfloor \log_a n \rfloor -1}
a^j \times (\lfloor \log_a n \rfloor - j)
(a^{\lfloor \log_a n \rfloor + 1 -j} -1)$$
which turns into
$$a^{\lfloor \log_a n \rfloor}
+ \sum_{j=0}^{\lfloor \log_a n \rfloor -1}
(\lfloor \log_a n \rfloor - j)
(a^{\lfloor \log_a n \rfloor + 1} - a^j).$$
The sum produces four terms.
The first is
$$\lfloor \log_a n \rfloor^2
a^{\lfloor \log_a n \rfloor + 1}.$$
The second is
$$- \lfloor \log_a n \rfloor
\frac{a^{\lfloor \log_a n \rfloor}-1}{a-1}.$$
The third is
$$- \frac{1}{2} a^{\lfloor \log_a n \rfloor + 1}
(\lfloor \log_a n \rfloor -1) \lfloor \log_a n \rfloor$$
and the fourth is
$$\frac{1}{(a-1)^2}
\left(a + a^{\lfloor \log_a n \rfloor}
(\lfloor \log_a n \rfloor (a-1) -a)\right).$$
This bound represented by these four terms plus the leading term is
actually attained and cannot be improved upon. For the asymptotics we
only need the dominant term, which is
$$\left(a - \frac{1}{2} a \right)
\lfloor \log_a n \rfloor^2
a^{\lfloor \log_a n \rfloor}
= \frac{1}{2} a \lfloor \log_a n \rfloor^2
a^{\lfloor \log_a n \rfloor}.$$
Now for the lower bound, which occurs with a one digit followed by
zeroes to give
$$T(n) \ge a^{\lfloor \log_a n \rfloor}
+ \sum_{j=0}^{\lfloor \log_a n \rfloor -1}
a^j \times (\lfloor \log_a n \rfloor - j) \times
a^{\lfloor \log_a n \rfloor-j}.$$
This simplifies to
$$a^{\lfloor \log_a n \rfloor}
+ \sum_{j=0}^{\lfloor \log_a n \rfloor -1}
(\lfloor \log_a n \rfloor - j) \times
a^{\lfloor \log_a n \rfloor}$$
which is
$$a^{\lfloor \log_a n \rfloor}
+ a^{\lfloor \log_a n \rfloor}
\sum_{j=0}^{\lfloor \log_a n \rfloor -1}
(\lfloor \log_a n \rfloor - j)$$
which finally produces
$$a^{\lfloor \log_a n \rfloor}
+ a^{\lfloor \log_a n \rfloor}
\sum_{j=1}^{\lfloor \log_a n \rfloor} j$$
or
$$a^{\lfloor \log_a n \rfloor}
+ \frac{1}{2}
\lfloor \log_a n \rfloor (\lfloor \log_a n \rfloor +1)
a^{\lfloor \log_a n \rfloor}.$$
The dominant term here is
$$\frac{1}{2}
\lfloor \log_a n \rfloor^2 a^{\lfloor \log_a n \rfloor}.$$
Joining the dominant terms of the upper and the lower bound we obtain
the asymptotics
$$\color{#00A}{\lfloor \log_a n \rfloor^2 \times
a^{\lfloor \log_a n \rfloor}
\in \Theta\left((\log_a n)^2 \times a^{\log_a n}\right)
= \Theta\left((\log n)^2 \times n\right)}.$$
This MSE link has a series of similar calculations.
This was already answered multiple times on the site but here we go. Let $S(k)=2^{-k}T(2^k)$, then $S(k)=S(k-1)+\Theta(k)$, hence $S(k-1)+Ak\leqslant S(k)\leqslant S(k-1)+Bk$ for some finite $A$ and $B$. Iterating this yields $S(0)+A\sum\limits_{i=1}^ki\leqslant S(k)\leqslant S(0)+B\sum\limits_{i=1}^ki$, hence $S(k)=\Theta(k^2)$.
The final step, which cannot be made rigorous without some further hypothesis (such as, that the sequence $(T(k))_k$ would be nondecreasing), is to assert that $T(2^k)=\Theta(2^kk^2)$ implies that $T(n)=\Theta(n(\log n)^2)$.
Best Answer
You are correct that this can be solved via the Master Theorem, though probably not by the version you know. This version of the Master Theorem is more general and states, in part, that if you have a recurrence of the form $$ T(n)=aT(n/b)+f(n) $$ with $a\ge 1, b>1$ (which we have, since $a=4\ge1$ and $b=(3/2)>1$) then if there is a constant $\epsilon>0$ for which $$ f(n)=O(n^{\log_b a -\epsilon}) $$ then $T(n) = \Theta(n^{\log_b a})$.
To see that this is the case we're interested in, note that $\log_{3/2}4\approx3.419$ so if we can establish that there is an $\epsilon>0$ such that $$ f(n) = n^3\log n = O(n^{\log_b a -\epsilon}) $$ then it will follow that $T(n) = \Theta(n^{\log_{3/2} 4})$.
I presume you know that $\log n$ grows asymptotically slower than and positive power of $n$, meaning that for any $\alpha >0$ we'll have $\log n=O(n^\alpha)$ so $$ n^3\log n=O(n^{3+\alpha}) $$ For no particular reason, we'll take $\alpha = 0.4$ then we'll have $n^3\log n=O(n^{3.4})$ and so we'll have $$ f(n)=O(n^{3.4}) = O(n^{3.419-.019})=O(n^{\log_{3/2} 4-\epsilon}) $$ for $\epsilon\approx 0.019$. The conditions of this case of the Master Theorem are satisfied, so $$ T(n)=\Theta(n^{\log_{3/2}4}) $$