[Math] Solving $Ax=b$ under $L_1$ $|Ax-b|$ minimization

Question

I would like to solve a linear system $Ax = b$ under the $L_1$ norm constraint $\min(|Ax-b|)$.
All that I can find about $L_1$ minimization is a way to minimize $|x|_1$ subject to $Ax=b$.

I wanted to use linear programming in matlab to solve this problem. This lead me to solving a new system with linprog in matlab:

$$My = k$$

So I did some transformations, knowing that Linprog can solve the following problem:

$$\min f(x) \ \ \textrm{s.t.} \ \ Ax \leq b$$

I want to minimize this problem:

$$\min ||My – k||_1$$

for $y$

To convert this to the linprog form, I introduced a new vector I want to minimize, $z$:

$$\min \sum z$$

s.t.

$$My – k \leq z$$
$$-My + k \leq z$$

However, the constraints include both the variable $y$ and the constants $k$. We can formulate this in the required form by creating a bigger system:

$Ax \leq b$, where the matrix $A$ is:

$$\begin{pmatrix}
M & -I \\
-M & I
\end{pmatrix}
$$

the vector $x$ is equal to $y$ and $z$ stacked:
$$\begin{pmatrix}
y \\
z
\end{pmatrix}
$$

and the constraints are $k$ and $-k$ stacked:
$$\begin{pmatrix}
k \\
-k
\end{pmatrix}
$$

The problem is that this doesn't work. (Matlab can't find a good solution). I would like to know if there is another way to solve this problem? If you have heard about a matlab code that already does it?

Thanks a lot

Answer 1

If you have min $\|Ax-b\|_1$ where $A$ is $n\times m$ for $n>m$ and $b$ is $n\times 1$ (so the sum of the absolute values of all the components of the result vector $y=Ax-b$ where y is $n\times 1$) then you can solve this as an LP with linprog with a command like this:

linprog([zeros(m,1);ones(n,1)],[+A,-eye(n);-A,-eye(n)],[b;-b])

which is just a translation into Matlab of the usual trick described on Wikipedia.

Here is a polynomial fitting example that compares least absolute deviations to linear least squares.

clc
x = [0:1/100:1]';

%random polynomial of degree 4
poly = [7/960;-1/8;227/320;-23/15;1]/0.0583;
A = [x.^0,x.^1,x.^2,x.^3,x.^4];
b = A*poly;

% bigger k means less outliers. k = 2 has too many outliers
% for the least absolute deviations solution to lie on the initial
% polynomial
k = 3;
b(k*[1:101/k]) = x(k*[1:101/k]);

% linear least squares
poly = A\b;
y = A*poly;

% least absolute deviations
[n,m] = size(A);
poly = linprog([zeros(m,1);ones(n,1)],[+A,-eye(n);-A,-eye(n)],[b;-b]); poly = poly(1:m);
z = A*poly;

% blue lines: original data
% green lines: least squares solution
% magenta crosses: least absolute deviations solution
% red circles: outliers
plot(x,b,'b-',x,y,'g-',x,z,'m+',x(k*[1:101/k]),x(k*[1:101/k]),'ro')

% check least absolute deviations
check = fminsearch(@(x) norm(A*x-b,1), poly);
[[poly; norm(A*poly-b,1)], [check; norm(A*check-b,1)]]

This is what the output looks like