More of a question about notation of the Taylor expansion in Logistic regression that anything else.
Given a nonlinear cost function
$
J(\theta)= – \frac{1}{n} \sum_{i=1}^n y_i (b+x_iw) – ln(1+e^{(b+x_iw)})
$
and given a Taylor expansion to approximate the cost function
$
f(\theta) \approx f(\theta^{(t)})+f^{\prime}(\theta^{(t)}) (\theta^{(t+1)}-\theta^{(t)})
$.
What I want to know is if the constant term is supposed to be a derivative of the cost function
$
f(\theta) = \frac{d}{d\theta} J(\theta) = J^{\prime}(\theta)
$
or is the constant term just equal to the cost function
$
f(\theta) =J(\theta)
$
Best Answer
In general for Taylor expansion series, the constant term should be close to the value you are trying to approximate.
Therefore, if you are trying to approximate $J(\theta^{(t+1)})$, then the first term of your expansion should take a known value of $J$, such as the $J(\theta^{(t)})$.
If you are trying to approximate $J'(\theta^{(t+1)})$, then the first term should be $J'(\theta^{(t)})$.