I think that you might as well replace
t = dctmtx(8);
y = blkproc(x, [8 8], 'P1 * x * P2', t, t');
with
y = blkproc(x, [8 8], 'dct2(x)')
and the quality is best for small values and very few coefficients are kept for larger values like 20. I hope this helps.
Best Answer