Python Numpy Calculus – Gradient of a Multivariate Function

calculusgradientnumpypython

I'm trying to calculate the gradient of multivariate function g using NumPy.

g = lambda w: -np.sin(np.pi*np.sum(w**2)) + np.log(np.sum(w**2))

gradient = lambda w: ...

the parameter w is a vector, for example, w = np.array([0.5,0.5]). I calculated the gradient like this;
gradient = lambda w: np.pi*np.sum(2*w)*-np.cos(np.pi*np.sum(w**2)) + np.sum(2*w)*(1/np.sum(w**2))

It does not give meaningful results. Is this formula correct or not?

Best Answer

If we write it more explicitly, $$f(w)=-\sin\left(\pi\sum w_i^2\right)+\ln \left(\sum w_i^2\right)$$

$${\partial f \over \partial w_i}=-\cos\left(\pi\sum w_i^2\right)\pi2w_i+\frac{2w_i}{\sum w_i^2}$$

So, the derivative is a vector; which means your expression will look like

gradient = lambda w: -np.pi*2*w*np.cos(np.pi*np.sum(w**2)) + 2*w*(1/np.sum(w**2))

when used denominator layout.

Related Solutions

Solved – Gradient of loss function for (non)-linear prediction functions

The chain rule is $$ (f(g(x)))' = f'(g(x))g'(x). $$ In a multivariate setting, with $f:\mathbb{R}\rightarrow\mathbb{R}$ and $g:\mathbb{R}^p\rightarrow\mathbb{R}$ this becomes $$ \nabla f(g(w)) = \nabla g(w)f'(g(w)). $$ Remember that the chain rule is $$ f(g(x)) = \frac{\partial z}{\partial x}\frac{\partial f}{\partial z}, $$ where $z = g(x)$.

Thus, if we let $$ L(w;X,y) = f(y, \text{pred}(X, w)) := f(y - \text{pred}(X, w)) = \|y - \text{pred}(X, w)\|_2^2 = \|y - Xw\|_2^2, $$ then $$ f(z) = \|z\|_2^2 = z^tz $$ and $$ z = g(w) = y - \text{pred}(X, w) = y - Xw. $$ We have, $$ \nabla f(z) = \nabla z^tz = 2z $$ and $$ \nabla g(w) = \nabla y - Xw = X^t. $$

Hence, the gradient of your linear estimator is $$ \nabla L(w;x,y) = \nabla g(w)\nabla f(z) = X^t\cdot2z = X^t\cdot2(y - Xw). $$

An example of a non-linear prediction function that has a gradient that is not the feature vector x would be e.g. logistic regression.

Solved – Implementing Lasso Regression in Numpy

As @verdery and @Matthew Drury wrote in the comment above, there is no analytical expression for Lasso Regression like there is for Ordinary Least Square and Ridge Regression. Anyway, I tested your Ridge Regression $\hat{\beta}$ code

np.linalg.inv(X.T @ X + alpha * np.identity(X.shape[1]) @ (X.T @ y)

I used an X-matrix of shape (1600,12). What happened is that $\hat{\beta}$ (which is supposed to be a (12,1) shaped array/vector) became a (12,12) shaped matrix. Then I did a change in the code and wrote it like this instead:

np.linalg.inv(X.T.dot(X) + alpha * np.identity(X.shape[1])).dot(X.T).dot(y)

Now I got the right (12,1) shape of my $\hat{\beta}$ vector

Best Answer

Related Solutions

Solved – Gradient of loss function for (non)-linear prediction functions

Solved – Implementing Lasso Regression in Numpy

Related Question