I know the cross entropy function can be used as the cost function, if the activation function is logistic function: i.e.: $\frac{1}{1 + e^{-x}}$
However, I just wonder:
Can the cross entropy cost function be used with many other activation functions, such as tanh?
Best Answer
Yes we can, as long as we use some normalizor (e.g. softmax) to ensure that the final output values are in between 0 and 1 and add up to 1.
If you're doing binary classification and only use one output value, only normalizing it to be between 0 and 1 will do.
Update
As mentioned by Sycorax, depending on what procedure you use to shifting and rescaling tanh back to [0,1], you could end up with a logistic unit again, see this answer.