MATLAB: Fitrsvm doesn’t vectorize the custom kernel

custom kernelfitrsvm

I'm working on an iterative function approximation algorithm. As part of that algorithm I use fitrsvm to update a kernel regression each iteration. This all works great when I use MATLAB's built in kernels. However, once I venture outside of MATLAB's kernels and start using my own the runtime of fitrsvm increases by nearly 4 times. That is, the runtime increases when I call:

m = fitrsvm(X, Y, 'KernelFunction','my_kernel');

To try to understand why I placed a break point in my_kernel. By doing this it appears that fitrsvm is not taking advantage of vectorization. The arguments being passed into my_kernel by fitrsvm behave as if it is in a loop similar to what I have below:

%how my_kernel appears to be called in fitrsvm based on the arguments passed to my_kernel
%though I admit I can't actually see the code since it seems to dispatch to a C file
for i = 1:size(X,1):
    for j = 1:size(X,1):
        G(i,j) = feval('my_kernel',X(i,:), X(j,:))
    end
end

This same inefficient behavior also occurs when I predict the responses:

response = predict(m,Z)

I've managed to get a considerable speed up by replacing predict with the following:

response = m.Bias + m.Alpha' * feval('my_kernel',m.SupportVectors,Z);

Has anybody else noticed this behavior? Any suggestions? Am I missing a parameter maybe? I'm working on 2020a.

Best Answer

I had this annoying problem too. I reached the conclusion that the best way to solve it is to use a 'precompiled' kernel and hack the kernel function as follows:

function g = precomp(u,v)
global K
g = K(u,v);
end

Where u and v correspond to the indexes of the observations. To achieve such indexing, I replaced the training data with the indexes as follows:

global K
K = myvectorizedkernelfcn(X,X);
Xidx = 1:size(X,1); % This corressponds to the number of rows
mdl = fitrsvm(Xidx',Y,'KernelFunction','precomp');

I hope this helps. Cheers.

Related Solutions

MATLAB: How to prune the weak learners in an ensemble learner

I'm answering my own question. (Actually, I had found the answer before asking, after much fiddling, but I thought it might be useful for others.)

NOTE: this is a somewhat disgusting hack. I'd love to know if there is a better way!

Though the "Trained" property of my ensemble is set-protected, and the constituent CompactClassificationTrees don't seem to possess a publicly accessible "prune" method, the hidden "Impl" property of each is publicly set-able and allows me to do what I want. So after using a template tree with pruning on to create an initial ensemble, I was able to loop over the resulting set of trained trees and prune each like the following. (The key is the two usage of "Impl" to access public implementations I could adjust.)

tTree = templateTree( ... , 'Prune', 'on');
ensembleCTrees = fitensemble(xdata, labels, 'AdaBoostM2', 10, tTree);
prunedEnsemble = ensembleCTrees;
for iTree = 1:maxTrees
  prunedEnsemble.Impl.Trained{iTree}.Impl = prune( ...
     prunedEnsemble.Impl.Trained{iTree}.Impl, 'Level', 20);
end

Then, I could try different settings of the prune level and compare prediction performance between the original and pruned ensembles.

MATLAB: Matérn kernel functions for support vector regression

The KernelFunction argument accepts custom function. You can create a function for matern kernel, and pass it to KernelFunction argument.

However, OptimizeHyperparameters only selects one of the inbuilt Kernel functions that are provided. So it's not possible to provide a custom function to OptimizeHyperparameters.

As a workaround you can use Bayesian optimization. Consider the following example,

load carsmall.mat
X = [Horsepower,Weight];
Y = MPG;
% This is how you pass custom KernelFunction
mdl = fitrsvm(X, Y, 'KernelFunction', 'myKernel'); 
% List out the functions you want to test
kernel = optimizableVariable('kernel', {'myKernel', 'myKernel2'}, 'Type', 'categorical');
% Define the function to optimize. Typically this will be
% the loss function on test dataset. For simplicity, I have passed the training data
fun = @(x)loss(fitrsvm(X, Y, 'KernelFunction', char(x.kernel)), X, Y);
% Run bayesian optimization
results = bayesopt(fun, [kernel]);

myKernel and myKernel2 are defined separately. You can define the required functions.

function G = myKernel(U,V)
    [m, ~] = size(U);
    [n, ~] = size(V);
    G = ones([m,n]);
end

And,

function G = myKernel2(U,V)
    [m, ~] = size(U);
    [n, ~] = size(V);
    G = zeros([m,n]);
end

Note that we define multiple kernel functions as kernel function itself does not take arguments except U and V.

Related Question