artificial intelligenceneural networks
In an artificial neural network, neurons take inputs from their previous layer, transform this signal using an activation function (in my case, the sigmoid function), and send this transformed signal along weights to the neurons in the next layer.
Do first layer neurons use an activation function to process the raw data they are fed, or do they send the raw signal along weights to the next layer without any activation function?
JunJun yes that is correct - everything stays the same between training and test - you just take maximum to determine the classification. However, rather than a sigmoid on each class output, you might consider (training/testing) with softmax function on output layer: which is giving you 'true' probability in each class ( outputs sum to 1) Softmax
Maybe what makes you think of ReLU is the hinge loss $E = max(1-ty,0)$ of SVMs, but the loss does not restrict the output activation function to be non-negative (ReLU).
For the network loss to be in the same form as SVMs, we can just remove any non-linear activation functions off the output layer, and use the hinge loss for backpropagation.
Moreover, if we replace the hinge loss with $E = ln (1 + exp(−ty))$ (which looks like a smooth version of hinge loss), then we'll be doing logistic regression as typical sigmoid + cross-entropy networks. It can be thought of as moving the sigmoid function from the output layer to the loss.
So in terms of loss functions, SVMs and logistic regression are pretty close, though SVMs use a very different algorithm for training and inference based on support vectors.
There's a nice discussion on the relation of SVM and logistic regression in section 7.1.2 of the book Pattern Recognition and Machine Learning.
Best Answer
To answer your question directly, the first layer is fed the raw input and weights which are then transformed via the activation function at the subsequent neurons.
The activation function is applied at each neuron not between neurons. The weights are multiplied by the prior layers outputs and summed for each neuron and then transformed via the activation function.
$Z^{(l+1)} = g(A^{(l)}\dot{}W^{(l)})$
where
$Z^{(l+1)}$ = output for next neuron layer ($l$)
$g$ = activation function
$A^{(l)}$ = previous layer data ($l$)
$W^{(l)}$ = weights for given layer