[Math] Operator Norm of a Linear Transformation

functional-analysisnormed-spacesreal-analysis

PROBLEM For the linear transformation $T:\mathbb{R}^n\rightarrow\mathbb{R}^m$ equipped with the $l^1$-norm, namely, for $x\in\mathbb{R}^n$, $||x||=\sum_{j=1}^n |x_j|$ and similarly for $x\in\mathbb{R}^m$, the operator norm is

$||T||_{op}=\max_{1\leq j\leq n}\sum_{i=1}^m |T_{ij}|$

Where the transform is defined to be matrix multiplication on left by matrix $T\in\mathbb{R}^{m\times n}$.

My problem is that I don't understand how this follows from the definition of the operator norm, namely that $||T||_{op}=\sup_{||x||=1}||Tx||$, where I've tried to write down a proof.

Work so far: Not very far (I don't agree with statement intuitively). Let $x\in\mathbb{R}^n$, $y\in\mathbb{R}^m$ and $T\in\mathbb{R}^{m\times n}$. Following the definition of an operator norm,

$||T||_{op}=\sup_{||x||=1}||Tx||=\sup_{||x||=1}\sum_{i=1}^m|y_i|$ where $y_i=\sum_{j=1}^n T_{ij}x_j$.

Then $||T||_{op}=\sup_{||x||=1}\sum_{i=1}^m|\sum_{j=1}^n T_{ij}x_j|$

This is where I have no clue how to proceed. Is this the right direction, or should I try another approach?

Best Answer

Go ahead. Use triangle inequality, and interchange the summation. Then you find $||T||_{op}\le\max_{1\leq j\leq n}\sum_{i=1}^m |T_{ij}|$. Let the max be attained at $j=j_0$. Now choose $x$ to be $e_{j_0}$.

Related Solutions

[Math] Operator Norm of a Linear Transformation of a Matrix

To some extent, the operator norm is just a way to define a useful structure on the set of linear operators. And, as you've already mentioned, this structure resembles usual Euclidean space: you can add and subtract two operators, multiply them by scalar and measure "how big" is this operator. This is just called a normed vector space. Why one might need this sort of structure will be revealed later when the author will speak about proofs of uniqueness and existence theorems for solutions of ODEs.

But let's get back to your questions.

However, I still don't quite understand what the operator norm of a linear transformation is or what it's purpose it (other than used to define the concept of convergence in a linear space).

Hey, don't deny this purpose, it's a very useful and important one :)
Operator norm also can be used when studying numerical methods, e.g. the concept of condition number can be described using operator norms.

Speaking about what is the operator norm, there are (at least) 3 equivalent definitions of operator norm:

$\| T \| = \max\limits_{x \neq 0} \; \frac{\vert Tx \vert}{\vert x \vert}$
$\| T \| = \max\limits_{\vert x \vert \leqslant 1} \; \vert Tx \vert$
$\| T \| = \max\limits_{\vert x \vert = 1} \; \vert Tx \vert$

So the first definition compares the norm of the image $Tx$ with norm of $x$ among all non-zero vectors of vector space. Hence operator norm means maximal relative stretch.

Last two definitions can be perceived more geometrically. The set $\vert x \vert \leqslant 1$ is a closed unit ball in vector space. Its image is not a ball, it might be stretched along some axis and contracted along others or even transformated in more complicated manner (eigenvalues to the rescue!). Operator norm tells you a closed ball of what size would be enough to contain the whole image of unit ball — and its size is determined by the norm of farthest point of unit ball image. In definition three you just look at the image of unit sphere and everything else is the same.

What stumps me even more is trying to compute the operator norm of any linear transformation, for example

\begin{array} d \begin{bmatrix} 1&0\\5&1\\ \end{bmatrix} \end{array}

I can offer you a simple trick here. There are few ways how to compute matrix norm in low-dimensional cases like 2 or 3 (using the third definition of matrix norm, it'll be a not so hard optimization problem of dimension 1 or 2), but they are not of great utility.

So, we want to find a norm of some matrix $A$. Since we're using standard Euclidean norm, $\vert Ax \vert = \sqrt{(Ax, Ax)}$, where $( \cdot\, , \cdot )$ is standard dot product in $\mathbb{R}^n$. So $(Ax, Ax)$ equals $(Ax)^{\rm T}(Ax) = x^{\rm T} (A^{\rm T}A) x $. The matrix $A^{\rm T}A$ is symmetric and positive semi-definite. We now have that $\| A \| = \max\limits_{\vert x\vert =1} \; \sqrt{x^{\rm T} (A^{T}A) x} $. The solution to the problem $\max\limits_{\vert x\vert =1} \; x^{\rm T} (A^{\rm T}A) x $ is well-known (just replace "Hermitian" with "symmetric" and everything will be fine). So, the task of finding operator norm subordinate to standard Euclidean norm for matrix $A$ is the same as the finding largest eigenvalue of $A^{\rm T}A$ (and it's always guaranteed to be non-negative since $A$ is symmetric and positive semi-definite).

[Math] Show that $T$ in $\ell^2(\mathbb{N})$ is a bounded linear operator and compute the operator norm $\|T\|$

This question is Old but I reckon its still worth answering since I can't find any other related questions. We had this question in a Hilbert space course I am currently doing at Uni. It is in a slightly different form but it should be more or less the same for this question.

Proposition.

let $(a_n)_{n \in \mathbb{N}}\subset \mathbb{C}$ be a fixed bounded sequence. Then the operator \begin{align*} T:&\ell_2(\mathbb{N},\mathbb{C}) \rightarrow \ell_2(\mathbb{N},\mathbb{C})\\ &(x_n) \mapsto(a_nx_n). \end{align*} Is a bounded linear operator and $||T||_{\text{op}}=\sup_{n \in \mathbb{N}}|a_n|$.

Proof:

Throughout this proof we will endow $\ell_2$ with the usual 2-norm.

We start with Linearity.

let $\lambda \in \mathbb{C}$ and $(x_n)_{n \in \mathbb{N}},(y_n)_{n \in \mathbb{N}}\in \ell_2(\mathbb{N},\mathbb{C})$ then we compute; \begin{align*} T(\{\lambda x_n+y_n\}_{n \in \mathbb{N}})&= \{a_n(\lambda x_n+y_n)\}_{n \in \mathbb{N}} & \text{[Definition of $T$]}\\ &=\{a_n\lambda x_n+a_ny_n\}_{n \in \mathbb{N}}\\ &=\{\lambda a_n x_n\}_{n \in \mathbb{N}}+\{a_ny_n\}_{n \in \mathbb{N}} & \text{[Point wise addition]}\\ &=\lambda \{a_n x_n\}_{n \in \mathbb{N}}+\{a_ny_n\}_{n \in \mathbb{N}}\\ & = \lambda T(\{x_n\}_{n \in \mathbb{N}})+T(\{y_n\}_{n \in \mathbb{N}}). \end{align*} Thus we can see that $T$ is linear.

Next we show that $T$ is bounded.

Let $(x_n)_{n \in \mathbb{N}},\in \ell_2(\mathbb{N},\mathbb{C})$ be a arbitrary sequence. Then we compute; \begin{align*} ||T(x_n)||_2 = ||(a_nx_n)||_2 = \sqrt{\sum_{i=1}^{\infty}|a_nx_n|^2}. \end{align*} Now The boundedness of $(a_n)$ becomes crucial. Since $(a_n)$ is bounded we can find $\sup_{n \in \mathbb{N}}|a_n|$ (if we knew more about $(a_n)$ we could find this exactly). Now let $M = \sup_{n \in \mathbb{N}}|a_n|$, then we have \begin{align*} ||T(x_n)||_2 = \sqrt{\sum_{i=1}^{\infty}|a_nx_n|^2} \leq. \sqrt{\sum_{i=1}^{\infty}|Mx_n|^2} = M\sqrt{\sum_{i=1}^{\infty}|x_n|^2} = M||(x_n)||_2. \end{align*} Since $(x_n)$ was arbitrary, this is true for all sequences in $\ell_2$. Consequently $T$ is bounded.

Now finally we find $||T||_{\text{op}}$: Recall that $||T||_op$ is the minimum of the bounds of $T$. To show that $||T||_{\text{op}}=\sup_{n \in \mathbb{N}}|a_n|$ we need to show the following;

If $M \geq 0$ is a finite constant such that \begin{align*} ||T(x_n)||_2 \leq M||(x_n)||_2\ \ \ \ \ \ \ \ (1) \end{align*} for all $(x_n)\in \ell_2$ then $\sup_{n \in \mathbb{N}}|a_n| \leq M$. So we fix $M \geq 0$ that satisfies $(1)$. Now for all $(x_n) \in \ell_2$ we have \begin{align*} ||T(x_n)||_2 = \sqrt{\sum_{i=1}^{\infty}|a_nx_n|^2} \leq M\sqrt{\sum_{i=1}^{\infty}|x_n|^2} = M||(x_n)||_2. \end{align*} Now this holds in particular when $||(x_n)||_2 =1$. For every $n \in \mathbb(N)$ we define the sequence \begin{align*} x_n &= (x_{n,k})_{k \in \mathbb{N}}\\ x_{n,k} &:= \delta_{k,n}. \end{align*} where $\delta_{k,n}$ is the Kronecker delta. Then we have \begin{align*} ||x_n||_2 = \sqrt{\sum_{i=1}^{\infty}|x_{n,i}|^2}=\sqrt{|x_{n,n}|^2} = 1 \end{align*} thus $x_n \in \ell_2$. Then we have \begin{align*} Tx_n &= (a_k x_{n,k})_{k \in \mathbb{N}}\\ Tx_{n,k} &:= a_k \delta_{k,n}. \end{align*} Now we also have \begin{align*} ||Tx_n||_2 = \sqrt{\sum_{i=1}^{\infty}|a_kx_{n,k}|^2}=\sqrt{|a_n|^2} =|a_n|. \end{align*} then by the way we chose $M$ we have \begin{align*} ||Tx_n||_2 \leq M||x_n||_2 \implies |a_n| \leq M. \end{align*} However remember that there was nothing special about $n$ so that $|a_n| \leq M$ for all $n \in \mathbb{N}$ and in particular this means $\sup_{n \in \mathbb{N}}|a_n| \leq M$, which proves the result.

Best Answer

Related Solutions

[Math] Operator Norm of a Linear Transformation of a Matrix

[Math] Show that $T$ in $\ell^2(\mathbb{N})$ is a bounded linear operator and compute the operator norm $\|T\|$

Related Question