Ch 3.5. Vector spaces, subspaces, rank, and more

• Prev: matrix-inverse-linear-systems
• Next: matrix-decompositions

In progress - Rank and non-square matrix explanation

1. Spans and spaces

• Span: The span of a set of vectors is all linear combinations of those vectors
• Vector space is denoted as ${\mathbb{R}}^n$.
  • Every element in a vector space can be written as a linear combination of the elements in the basis (unit) vectors
    • Basis (unit) vectors (example): For a 2D vector space, the basis vectors are $\hat{i}=\left[\begin{array}{c}1\\ 0\end{array}\right]$ and $\hat{j}=\left[\begin{array}{c}0\\ 1\end{array}\right]$
    • A matrix $\text{A}$ applies a linear transformation to a vector space (i.e. all vectors in the space)
    • The columns of $\text{A}$ represent the landing points for the basis (unit) vectors after the transformation
      • By extension, $\text{A}$ moves every input vector (more precisely, the point where every vector’s tip is) linearly to a new location.
    • We only need to know how $\text{A}$ transforms the bases $\hat{i}$ and $\hat{j}$, since
      • any other vector $\text{v}$ is just a linear combination of $\hat{i}$ and $\hat{j}$ both before and after being transformed by $\text{A}$
• Subspace: a subset of a larger vector space
  • Geometric intuition: A subspace of a 3D vector space is a 2D plane that passes through the origin
  • Column space (aka range, or image): Span (i.e. set of all possible linear combinations) of the column vectors of $\text{A}$
    • Geometrically: Essentially, the entire output space
  • Row space: Span of the row vectors of $\text{A}$
  • Null space (or kernel): If $\text{A}\cdot \text{x}=\text{0}$, the span of all solutions $\text{x}$ constitutes the null space of $\text{A}$
    • Geometrically: The space of all vectors that $\text{A}$ maps onto the zero vector (i.e. they are squished by $\text{A}$ onto the origin)
  • Left-Null space: If ${\text{A}}^{\text{T}}\cdot \text{x}=\text{0}$, the span of all solutions $\text{x}$ constitutes the left-null space of $\text{A}$

2. More properties

• Rank: The number of linearly independent columns (or rows) in $\text{A}$ is its rank
• Orthonormal vectors: Two unit length vectors whose inner (i.e. dot) products are 0 (e.g. $\hat{i}$ and $\hat{j}$)
• Real value matrices:
  • Orthogonal matrices: If $\text{A}$‘s rows and cols are orthonormal vectors, $\text{A}$ is an orthogonal matrix. It satisfies:
    • ${\text{A}}^{\text{T}}\text{A}={\text{AA}}^{\text{T}}=\text{I}$
  • Symmetric matrix: where ${\text{A}}^{\text{T}}=\text{A}$ (square matrices only)
• Complex value matrices
  • Hermitian matrix: Complex matrices’ analog to orthogonal matrix
  • Unitary matrix: Complex matrices’ analog to symmetric matrix
• Determinant: This can be computed for any square matrix $\text{A}$
  • Matrices are only invertible if $det(\text{A})\ne 0$. Such matrices are non-singular; and satisfy ${\text{AA}}^{-1}=\text{I}$
  • Matrices where $det(\text{A})=0$ are not invertible. They are singular matrices

3. Examples of the above (where relevant):

• The span of vectors $\left[\begin{array}{c}1\\ 0\end{array}\right]$ and $\left[\begin{array}{c}0\\ 1\end{array}\right]$ is the whole $x$-$y$ plane.
• A vector, $\text{v}$, with 3 elements is said to exist in vector space ${\mathbb{R}}^3$
• The ${\mathbb{R}}^3$ vector, $\text{v}$ is composed of (i.e. a linear combination of) the basis vectors $\hat{i}=\left[\begin{array}{c}1\\ 0\\ 0\end{array}\right]$, $\hat{j}=\left[\begin{array}{c}0\\ 1\\ 0\end{array}\right]$ and $\hat{k}=\left[\begin{array}{c}0\\ 0\\ 1\end{array}\right]$.
• The subspace of a 3D vector (in ${\mathbb{R}}^3$) is the span of vectors $\left[\begin{array}{c}1\\ 0\\ 0\end{array}\right]$, $\left[\begin{array}{c}0\\ 1\\ 0\end{array}\right]$.
  • in this case the 2D $x$-$y$ plane is a subspace (subset) of the 3D $x,y,z$ vector space
• Vectors $\left[\begin{array}{c}1\\ 2\\ 3\end{array}\right]$ and $\left[\begin{array}{c}10\\ 20\\ 30\end{array}\right]$ are linearly dependent since one is a multiple of the other.
  • A matrix with those two vectors would be rank 1

TODO: Kroenecker product, Tensors, Tensor products,

https://en.wikipedia.org/wiki/Tensor https://en.wikipedia.org/wiki/Tensor_product https://en.wikipedia.org/wiki/Kronecker_product https://en.wikipedia.org/wiki/Block_matrix https://math.stackexchange.com/questions/973559/outer-product-of-two-matrices https://stackoverflow.com/questions/24839481/python-matrix-outer-product

4. Back to Non-Square matrices (A geometric perspective)

A non-square matrix $\text{A}:\text{A}\in {\mathbb{R}}^{m\times n}$ has $m$ rows and $n$ columns. Geometrically, $\text{A}$ transforms a vector space ${\mathbb{R}}^n$ into a vector space ${\mathbb{R}}^m$.

For example, the matrix $\text{A}$ transforms a 2D vector space into a 3D vector space.

• Specifically, the two basis vectors (i.e. from 2D space) transform to 3D coordinates as follows:

$$\begin{array}{rl}\text{A}& =\left[\begin{array}{cc}3& 1\\ 4& 1\\ 5& 9\end{array}\right]\text{transforms: }\hat{i}:\left[\begin{array}{c}1\\ 0\end{array}\right]\to \left[\begin{array}{c}3\\ 4\\ 5\end{array}\right]\text{and}\hat{j}:\left[\begin{array}{c}0\\ 1\end{array}\right]\to \left[\begin{array}{c}1\\ 1\\ 9\end{array}\right]\end{array}$$

Similarly, matrix $\text{B}$ transforms a 3D vector space into a 2D vector space.

• Specifically, the three basis vectors (i.e. from 3D space) transform to 2D coordinates as follows:

$$\begin{array}{rl}\text{B}& =\left[\begin{array}{ccc}1& 2& 3\\ 4& 5& 6\end{array}\right]\text{transforms: }\hat{i}:\left[\begin{array}{c}1\\ 0\\ 0\end{array}\right]\to \left[\begin{array}{c}1\\ 4\end{array}\right]\text{,}\hat{j}:\left[\begin{array}{c}0\\ 1\\ 0\end{array}\right]\to \left[\begin{array}{c}2\\ 5\end{array}\right]\text{and}\hat{k}:\left[\begin{array}{c}0\\ 0\\ 1\end{array}\right]\to \left[\begin{array}{c}3\\ 6\end{array}\right]\end{array}$$

4.1. Rank

• Definition: The rank of $\text{A}:\text{A}\in {\mathbb{R}}^{m\times n}$ is the number of linearly independent columns or rows in $\text{A}$
  • Rank is the number of dimensions in the column-space (aka. span of the column vectors of $\text{A}$ in ${\mathbb{R}}^m$ (or ${\mathbb{R}}^n$)).
  • Consequently, the number of linearly independent columns in a matrix $\equiv$ the number of linearly independent rows in that matrix.

4.1.1. “Full Rank” Matrix

• $\text{A}$ is full rank if $\text{rank}(\text{A})=\min (m,n)$ (i.e. the rank is as high as it can be)
• This is the same as:
  • if all columns of $\text{A}$ are linearly independent

4.1.2. Geometric intuition for Rank::

• If matrix $\text{A}$ is a transformation, the rank is the number of dimensions in the output space. Examples:
  • If some matrix $\text{A}$ transformed some arbitrary vector space ${\mathbb{R}}^n$ into a 1D line in ${\mathbb{R}}^1$, then $\text{rank}(\text{A})=1$
  • If $\text{A}$ transformed some vector space ${\mathbb{R}}^n$ to a 2D plane in ${\mathbb{R}}^2$, then $\text{rank}(\text{A})=2$
• In general, if $\text{A}$ transforms some vector space ${\mathbb{R}}^n$ to ${\mathbb{R}}^m$, then $\text{rank}(\text{A})=m$

4.1.3. Geometric intuition, examples:

• A $2\times 2$ matrix is full rank (i.e. rank 2) if the output is a 2D space (${\mathbb{R}}^2$; same as the input space). $\text{det}(\text{A})\ne 0$
• A $3\times 3$ matrix is full rank (i.e. rank 3) if the output is a 3D space (${\mathbb{R}}^3$; same as the input space). $\text{det}(\text{A})\ne 0$. However,
  • If the $3\times 3$ matrix collapsed the input space to a 2D plane, then it is only rank 2 (NB: but it can collapse further)
  • If the $3\times 3$ matrix collapsed the input space to a 1D line, then it is only rank 1
• A $2\times 3$ matrix is full rank (i.e. rank 2) if the output is a 2D plane in ${\mathbb{R}}^3$ (same as the input space)

2.1.4. Augmented Matrix

• If vector $\text{y}$ is concatenated to matrix $\text{A}$, we say ”$\text{A}$ augmented with $\text{y}$“. Denoted as $(\text{A}|\text{B})$.
  • if $\text{rank}((\text{A}|\text{B}))=\text{rank}(\text{A})+1$, then vector $\text{y}$ is “new” information
  • otherwise, if $\text{rank}((\text{A}|\text{B}))=\text{rank}(\text{A})$, it means $\text{y}$ can be created as a linear combination of the columns in $\text{A}$

# Compute the condition number and rank for matrix A = [[1,1,0],[0,1,0],[1,0,1]]
# If y = [[1],[2],[1]], get the augmented matrix [A,y]
 
from numpy import array, trace, concatenate
from numpy.linalg import cond, matrix_rank
 
A = array([[1, 1, 0], [0, 1, 0], [1, 0, 1]])
y = array([[1], [2], [1]])
print("A matrix's shape:", A.shape, "\ny vector's shape", y.shape)
 
print("\nCondition number(A):", cond(A))
print("Rank(A):", matrix_rank(A))
print("Trace(A):", trace(A))
 
A_y = concatenate((A, y), axis=1)
print("\nOriginal A matrix:\n", A, "\ny vector:\n", y, "\n\nAugmented (A|y) matrix:\n", A_y)
print("Rank(A_y):", matrix_rank(A_y))
 
print("\nNote that the rank of A and A_y are both 3, so y is a linear combination of the columns of A.")
print("Therefore, there is no new information in y that is not already in A.")
 

A matrix's shape: (3, 3) 
y vector's shape (3, 1)
 
Condition number(A): 4.048917339522305
Rank(A): 3
Trace(A): 3
 
Original A matrix:
 [[1 1 0]
 [0 1 0]
 [1 0 1]] 
y vector:
 [[1]
 [2]
 [1]] 
 
Augmented (A|y) matrix:
 [[1 1 0 1]
 [0 1 0 2]
 [1 0 1 1]]
Rank(A_y): 3
 
Note that the rank of A and A_y are both 3, so y is a linear combination of the columns of A.
Therefore, there is no new information in y that is not already in A.