What is a main difference between RBF neural networks and SVM with a RBF kernel, from a practical point of view?
Solved – a main difference between RBF neural networks and SVM with a RBF kernel
neural networkssvm
Related Solutions
Let $\mathcal{X}$ represent your input space i.e the space where your data points resides. Consider a function $\Phi:\mathcal{X} \rightarrow \mathcal{F}$ such that it takes a point from your input space $\mathcal{X}$ and maps it to a point in $\mathcal{F}$. Now, let us say that we have mapped all your data points from $\mathcal{X}$ to this new space $\mathcal{F}$. Now, if you try to solve the normal linear svm in this new space $\mathcal{F}$ instead of $\mathcal{X}$, you will notice that all the earlier working simply look the same, except that all the points $x_i$ are represented as $\Phi(x_i)$ and instead of using $x^Ty$ (dot product) which is the natural inner product for Euclidean space, we replace it with $\langle \Phi(x), \Phi(y) \rangle$ which represents the natural inner product in the new space $\mathcal{F}$. So, at the end, your $w^*$ would look like,
$$ w^*=\sum_{i \in SV} h_i y_i \Phi(x_i) $$
and hence, $$ \langle w^*, \Phi(x) \rangle = \sum_{i \in SV} h_i y_i \langle \Phi(x_i), \Phi(x) \rangle $$
Similarly, $$ b^*=\frac{1}{|SV|}\sum_{i \in SV}\left(y_i - \sum_{j=1}^N\left(h_j y_j \langle \Phi(x_j), \Phi(x_i)\rangle\right)\right) $$
and your classification rule looks like: $c_x=\text{sign}(\langle w, \Phi(x) \rangle+b)$.
So far so good, there is nothing new, since we have simply applied the normal linear SVM to just a different space. However, the magic part is this -
Let us say that there exists a function $k:\mathcal{X}\times\mathcal{X}\rightarrow \mathbb{R}$ such that $k(x_i, x_j) = \langle \Phi(x_i), \Phi(x_j) \rangle$. Then, we can replace all the dot products above with $k(x_i, x_j)$. Such a $k$ is called a kernel function.
Therefore, your $w^*$ and $b^*$ look like, $$ \langle w^*, \Phi(x) \rangle = \sum_{i \in SV} h_i y_i k(x_i, x) $$ $$ b^*=\frac{1}{|SV|}\sum_{i \in SV}\left(y_i - \sum_{j=1}^N\left(h_j y_j k(x_j, x_i)\right)\right) $$
For which kernel functions is the above substitution valid? Well, that's a slightly involved question and you might want to take up proper reading material to understand those implications. However, I will just add that the above holds true for RBF Kernel.
To answer your question, "Is the situation so that all the support vectors are needed for the classification?" Yes. As you may notice above, we compute the inner product of $w$ with $x$ instead of computing $w$ explicitly. This requires us to retain all the support vectors for classification.
Note: The $h_i$'s in the final section here are solution to dual of the SVM in the space $\mathcal{F}$ and not $\mathcal{X}$. Does that mean that we need to know $\Phi$ function explicitly? Luckily, no. If you look at the dual objective, it consists only of inner product and since we have $k$ that allows us to compute the inner product directly, we don't need to know $\Phi$ explicitly. The dual objective simply looks like, $$ \max \sum_i h_i - \sum_{i,j} y_i y_j h_i h_j k(x_i, x_j) \\ \text{subject to : } \sum_i y_i h_i = 0, h_i \geq 0 $$
To use RFE, it is a must to have a supervised learning estimator which attribute coef_ is available, this is the case of the linear kernel. The error you are getting is because coef_ is not for SVM using kernels different from Linear. It is in RFE documentation
A walk-around solution is presented in Feature selection for support vector machines with RBF kernel by Quanzhong Liu et. al.
Best Answer
A RBF SVM would be virtually equivalent to a RBF neural nets where the weights of the first layer would be fixed to the feature values of all the training samples. Only the second layer weights are tuned by the learning algorithm. This allows the optimization problem to be convex hence admit a single global solution. The fact that the number of the potential hidden nodes can grow with the number of samples makes this hypothetical neural network a non-parametric model (which is usually not the case when we train neural nets: we tend to fix the architecture in advance as a fixed hyper-parameter of the algorithm independently of the number of samples).
Off-course in practice the SVM algorithm implementations do not treat all the samples from the training set as concurrently active support vectors / hidden nodes. Samples are incrementally selected in the active set if they contribute enough and pruned as soon as their are shadowed by a more recent set of support vectors (thanks to the combined use of a margin loss function such as the hinge loss and regularizer such as l2). That allows the number of parameters of the model to be kept low enough.
On the other hand when classical RBF neural networks are trained with a fixed architecture (fixed number of hidden nodes) but with tunable input layer parameters. If we allow the neural network to have as many hidden nodes as samples, then the expressive power such a RBF NN would be much higher than the SVM model as the weights of the first layer are tunable but that comes at the price of a non convex objective function that can be stuck in local optima that would prevent the algorithm to converge to good parameter values. Furthermore this increased expressive power comes with a serious capability to overfit: reducing the number of hidden nodes can help decrease overfitting.
To summarize from a practical point of view:
RBF neural nets have a higher number of hyper-parameters (the bandwidth of the RBF kernel, number of hidden nodes + the initialization scheme of the weights + the strengths of the regularizer a.k.a weight decay for the first and second layers + learning rate + momentum) + the local optima convergence issues (that may or not be an issue in practice depending on the data and the hyper-parameter)
RBF SVM has 2 hyper-parameters to grid search (the bandwidth of the RBF kernel and the strength of the regularizer) and the convergence is independent from the init (convex objective function)
BTW, both should have scaled features as input (e.g. unit variance scaling).