I was hoping to understand what the smooth $l_1$ loss does, but I'm not able to find any good explanation of online, I know $l_1$ loss calculates the absolute error, but what is the use of smooth $L_1$, any answers would be helpful.
Solved – How to interpret smooth l1 loss
deep learningoptimizationregularization
Related Solutions
I think your network architecture is better understood as:
Gradients should flow back to the dimension reduction layer from both the PCA loss and the cross-entropy loss on the predictions. If you're using a graph-based system like Tensorflow, Theano, Caffe, etc., this should happen automatically. Note that your update for the PCA loss should actually be $$W_{t+1}=W_t + \eta \tilde{\Sigma}W_{t}$$ where $$\tilde{\Sigma}=E[{xx}^T]\approx\frac{1}{b}\sum^{b-1}_{i=0}{x_ix_i}^T$$which is just gradient ascent with the expectation taken over a mini-batch. Alternatively, you could precompute a covariance matrix using all (or a large portion of the very redundant) available data and use $\tilde{\Sigma}$ as a constant. So there is no need for a custom update.
However, let's assume you do want a custom update as represented by a custom gradient (for example, a reduced variance estimate of the gradient). Then you would only need to overwrite the gradient flowing back from the PCA loss into the reduction layer and most graph based programs make this fairly easy to do. If your custom update cannot be understood as an alternative gradient (although most can), then you could use ADMM where you alternate between optimizing with respect to the PCA loss using the custom update and applying gradient descent with respect to the cross-entropy loss. However, since you PCA loss is unbounded from above (unless there are some other constraints on the magnitude of $W$), I'd recommend against this.
Best Answer
Smooth L1-loss can be interpreted as a combination of L1-loss and L2-loss. It behaves as L1-loss when the absolute value of the argument is high, and it behaves like L2-loss when the absolute value of the argument is close to zero. The equation is:
$L_{1;smooth} = \begin{cases}|x| & \text{if $|x|>\alpha$;} \\ \frac{1}{|\alpha|}x^2 & \text{if $|x| \leq \alpha$}\end{cases}$
$\alpha$ is a hyper-parameter here and is usually taken as 1. $\frac{1}{\alpha}$ appears near $x^2$ term to make it continuous.
Smooth L1-loss combines the advantages of L1-loss (steady gradients for large values of $x$) and L2-loss (less oscillations during updates when $x$ is small).
Another form of smooth L1-loss is Huber loss. They achieve the same thing. Taken from Wikipedia, Huber loss is
$ L_\delta (a) = \begin{cases} \frac{1}{2}{a^2} & \text{for } |a| \le \delta, \\ \delta (|a| - \frac{1}{2}\delta), & \text{otherwise.} \end{cases} $