Least squares solutions of linear systems and orthogonal projections

Let A be an m x n matrix and V = Col(A) be the associated subspace of R^n.  For instance:

m=3
n=2
MatrixForm[A=Transpose[{{1,0,2},{0,1,-1}}]]

If b is a vector in R^n, then the system Ax = b has a solution if and only if b in Col(A).  Regardless, there is always a vector x in R^n such that minimizes ||Ax-b||.  This is called a least squares solution of Ax = b.

b={1,-1,1}

It turns out that x is a least squares solution if and only if it satisfies the so-called `normal equation' (A^T)A x = (A^T) b associated to the given system.

MatrixForm[ATA =Transpose[A].A]
 MatrixForm[ATb=Transpose[A].b]
x = {x1,x2}/.First[Solve[ATA.{x1,x2}==ATb,{x1,x2}]]

The underlying reason this works is that the vector x that we just found has the property that Ax is the orthogonal projection of b onto Col(A).

Show[Graphics3D[Line[{A.x,b}]],Graphics3D[{PointSize[Large],Point[A.x]}],Graphics3D[{PointSize[Large],Point[b]}],
ParametricPlot3D[A.{s,t},{s,-2,2},{t,-2,2}],PlotRange->{{-2,2},{-2,2},{-2,2}}]

It turns out that if the columns of A are independent (in which case they form a basis for Col(A)), then (A^T)A is invertible.  In this case we have the formula
	P = A ((((A^T) A))^(-1))(A^T)
for orthogonal projection onto Col(A).  This is one way to see that orthogonal projection onto a subspace is a linear transformation.

MatrixForm[P=A.Inverse[ATA].Transpose[A]]

P.{1,1,1}

Projection matrices P are distinguished by two special properties.  They are `symmetric’: P = (P^T) and `idempotent’: (P^(2 ))= P.

MatrixForm[P.P]

MatrixForm[Transpose[P]]