Linear Algebra – Applications of Systems of Linear Equations

linear algebra

Sorry if this questions is overly simplistic. It's just something I haven't been able to figure out.

I've been reading through quite a few linear algebra books and have gone through the various methods of solving linear systems of equations, in particular, $n$ systems in $n$ unknowns. While I understand the techniques used to solve these for the most part, I don't understand how these situations present themselves. I was wondering if anyone could provide a simple real-world example or two from data analysis, finance, economics, etc. in which the problem they were working on led to a system of $n$ equations in $n$ unknowns. I don't need the solution worked out. I just need to know the problem that resulted in the system.

Best Answer

One of the most frequent occasions where linear systems of $n$ equations in $n$ unknowns arise is in least-squares optimization problems. Let us look at an example. Let's say that we are studying two physical quantities $y$ and $x$ and we conjecture that $y$ is a second order polynomial function of $x$, i.e. $y=\alpha x^2 + \beta x + \gamma$ for some real numbers $\alpha$, $\beta$, $\gamma$ that are unknown. Let's say now that we perform experiments and obtain measurements $(x_1,y_1) \cdots (x_{100},y_{100})$. Applying the polynomial model on the measurements yields $y_i=\alpha x_i^2 + \beta x_i + \gamma$ for $i=1, \cdots 100$ or in matrix form $X k=y$ where $k=[\alpha \, \, \beta \, \, \gamma]^T$, $y=[y_1 \cdots y_{100}]^T$ and the $i^{th}$ row of $X$ is the row vector $[x_i^2 \, \, x_i \, \, 1]$. Now, as you might observe, we have $100$ equations in $3$ unknowns, i.e. our linear system $X k=y$ is overdetermined. Practically speaking, this system is consistent (i.e. it has a solution) only if indeed $y$ is related to $x$ via a second order polynomial equation (i.e. our conjecture is true) and additionally there is no noise in our measurements. So assume that none of the above two conditions is true. Hence the system $X k=y$ will not in general have a solution and one might consider finding a vector $k$ that instead minimizes $||X k - y||_2^2$, i.e. the square of the error. Then the solution of this optimization problem is the solution to the $3 \times 3$ system $X^T X k = X^T y$. This formulation comes up all the time in engineering, e.g. in signal prediction. So, least squares problems lead to square (i.e. $n \times n$) linear systems of equations.