Ch 1. Set Notation

• Next: vector-operations

Summary: Commonly encountered objects and calculations in Linear Algebra, implemented in Python.

1. Set Notation

• A collection of objects are denoted in braces $\{1,2,o_a,g\}$
  • Empty set (denoted $\{\}$ or $\varnothing$): the set containing no objects)
• Common set operations:
  • Union, $A\cup B$: Set containing all elements of $A$ or $B$
  • Intersection, $A\cap B$: Set containing all elements that belong to both $A$ and $B$
  • Proper/strict subset, $C\subset A$: $C$ is a strict subset of (i.e. included in; but is not equal to) $A$
    • Subset, $C\subseteq A$: $C$ is a subset of (i.e. included in; or is equal to) $A$
  • Proper/strict superset, $C\supset A$: $C$ is a strict superset of (i.e. includes; but is not equal to) $A$
    • Superset, $C\supseteq A$: $C$ is a superset of (i.e. includes; or is equal to) $A$
  • Relative complement:
  • Colon ($:$) means “such that”
  • $a\in A$ means “element $a$ is a member of set $A$”
  • Backslash ($\backslash$) means ""set minus” so if $a\in A$, then $A\backslash a$ means ”$A$ minus the element $a$”
• Some standard sets related to numbers:
  • Naturals: $\mathbb{N}=\{1,2,3,4,\cdots \}$
  • Wholes: $\mathbb{W}=\mathbb{N}\cup \{0\}$
  • Integers: $\mathbb{Z}=\mathbb{W}\cup \{-1,-2,-3,\cdots \}$
  • Rationals: $\mathbb{Q}=\{\frac{p}{q}:p\in \mathbb{Z},q\in \mathbb{Z}\backslash \{0\}\}$
  • Irrationals: $\mathbb{I}$ is the set of real numbers not expressible as a fraction of integers
  • Reals: $\mathbb{R}=\mathbb{Q}\cup \mathbb{I}$
  • Complex numbers: $\mathbb{C}=\{a+bi:a,b\in \mathbb{R},i=\sqrt{-1}\}$
• Example:
  • Let $S$ be the set of all real $(x,y)$ pairs such that $x^2+y^2=1$ Write $S$ using set notation:
  • $S=\{(x,y):x,y\in \mathbb{R},x^2+y^2=1\}$
• expand: Talk about real coordinate spaces

2. A note on Python syntax

• To define vectors and matrices in NumPy, use np.array([[]]) with double square brackets
  • A row vector: [[ csv row elements ]]
  • A column vector: [[elem 1], [elem 2], elem 3]], and you can put each on a new line
  • A matrix: [[ csv row 1], [ csv row 2], ...], and you can put each on a new line
• There are many ways to multiply matrices (with vectors or other matrices) in Python:
  • np.dot(A, B): matrix multiplication
  • np.matmul(A, B): matrix multiplication
    • @ operator: matrix multiplication (preferred in most cases)
  • np.inner(A, B): inner product
• Additional approaches for other specific operations:
  • np.cross(A, B): cross product (vectors)
  • np.multiply(A, B): element-wise multiplication (Hadamard)
  • np.outer(A, B): outer product
  • np.qr(A): QR decomposition
  • np.lu(A): LU decomposition
  • np.svd(A): SVD decomposition
  • np.frobenius(A, B): Frobenius inner product
  • np.vdot(A, B): dot product
  • np.einsum('ij,jk->ik', A, B): Einstein summation
  • np.tensordot(A, B): tensor product
  • np.kron(A, B): Kronecker product

import numpy as np
 
print("----------------- 4 python methods to find the dot product of two vectors -----------------")
# Dot product of two vectors
v = np.array([[10, 9, 3]])
w = np.array([[2, 5, 12]])
 
# The following python functions all calculate the dot product of two vectors
print("Vector dot product: v • w:")
print("py (preferred): np.dot(v, w.T) =", np.dot(v, w.T)[0][0])
print("py (method 2): np.inner(v, w) =", np.inner(v, w)[0][0])
print("py (method 3): np.matmul(v, w.T) =", np.matmul(v, w.T)[0][0])
print("py (method 4): v @ w.T =", (v @ w.T)[0][0])
 

----------------- 4 python methods to find the dot product of two vectors -----------------
Vector dot product: v • w:
py (preferred): np.dot(v, w.T) = 101
py (method 2): np.inner(v, w) = 101
py (method 3): np.matmul(v, w.T) = 101
py (method 4): v @ w.T = 101

print("\n----------------- 4 python methods to multiply matrix `A` by a vector `b` -----------------")
# Multiply matrix A by (column) vector b = [2, 3]
A = np.array([[1, 7], [2, 3], [5, 0]])
b = np.array([[2], [3]])  # column vector
print("A =\n", A, "\nb =\n", b)
 
# The following python functions all multiply matrix A by vector b
print("\nMatrix times vector: Ab")
print("py (preferred method): A @ b =\n", A @ b)
print("\npy (method 2): np.matmul(A, b) =\n", np.matmul(A, b))
print("\npy (method 3): np.dot(A, b) =\n", np.dot(A, b))
print("\npy (method 4) np.inner(A, b.T) =\n", np.inner(A, b.T))  # confusing that second arg needs transposing
 

----------------- 4 python methods to multiply matrix `A` by a vector `b` -----------------
A =
 [[1 7]
 [2 3]
 [5 0]] 
b =
 [[2]
 [3]]
 
Matrix times vector: Ab
py (preferred method): A @ b =
 [[23]
 [13]
 [10]]
 
py (method 2): np.matmul(A, b) =
 [[23]
 [13]
 [10]]
 
py (method 3): np.dot(A, b) =
 [[23]
 [13]
 [10]]
 
py (method 4) np.inner(A, b.T) =
 [[23]
 [13]
 [10]]

print("\n----------------- 4 python methods to pre-multiply matrix `A` by a vector `c.T` -----------------")
# Transpose the column vector c = [2, 3, 1], and multiply the resulting row vector c^T by matrix A
A = np.array([[1, 7], [2, 3], [5, 0]])
c = np.array([[2], [3], [1]])  # column vector
print("c.T =\n", c.T, "\nA =\n", A)
 
# The following python functions all pre-multiply matrix A by row vector c^T
print("\nVector times matrix (pre-multiply): c^T A")
print("py (preferred method): c.T @ A =", (c.T @ A)[0])
print("py (method 2): np.matmul(c.T, A) =", np.matmul(c.T, A)[0])
print("py (method 3): np.dot(c.T, A) =", np.dot(c.T, A)[0])
print("py (method 4): np.inner(c, A) =", np.inner(c.T, A.T)[0])  # confusing that second arg needs transposing
 

----------------- 4 python methods to pre-multiply matrix `A` by a vector `c.T` -----------------
c.T =
 [[2 3 1]] 
A =
 [[1 7]
 [2 3]
 [5 0]]
 
Vector times matrix (pre-multiply): c^T A
py (preferred method): c.T @ A = [13 23]
py (method 2): np.matmul(c.T, A) = [13 23]
py (method 3): np.dot(c.T, A) = [13 23]
py (method 4): np.inner(c, A) = [13 23]

print("\n----------------- 4 python methods to multiply matrix `A` by a matrix `B` -----------------")
A = np.array([[1, 7], [2, 3], [5, 0]])
B = np.array([[2, 6, 3, 1], [1, 2, 3, 4]])
print("A =\n", A, "\nB =\n", B)
 
# The following python functions all multiply matrix A by matrix B
print("\nMatrix times matrix: AB")
print("py (preferred method): A @ B =\n", A @ B)  # <-- inner dims match (p=2), so output is a [3x4] matrix
print("\npy (method 2): np.matmul(A, B) =\n", np.matmul(A, B))
print("\npy (method 3): np.dot(A, B) =\n", np.dot(A, B))
print("\npy (method 4): np.inner(A, B.T) =\n", np.inner(A, B.T))  # confusing that second arg needs transposing
 

----------------- 4 python methods to multiply matrix `A` by a matrix `B` -----------------
A =
 [[1 7]
 [2 3]
 [5 0]] 
B =
 [[2 6 3 1]
 [1 2 3 4]]
 
Matrix times matrix: AB
py (preferred method): A @ B =
 [[ 9 20 24 29]
 [ 7 18 15 14]
 [10 30 15  5]]
 
py (method 2): np.matmul(A, B) =
 [[ 9 20 24 29]
 [ 7 18 15 14]
 [10 30 15  5]]
 
py (method 3): np.dot(A, B) =
 [[ 9 20 24 29]
 [ 7 18 15 14]
 [10 30 15  5]]
 
py (method 4): np.inner(A, B.T) =
 [[ 9 20 24 29]
 [ 7 18 15 14]
 [10 30 15  5]]

Sources

• Python Numerical Methods: Basics of Linear Algebra
• https://en.wikipedia.org/wiki/Matrix_multiplication
• https://en.wikipedia.org/wiki/Dot_product
• https://en.wikipedia.org/wiki/Cross_product
• https://en.wikipedia.org/wiki/Hadamard_product_(matrices)
• https://en.wikipedia.org/wiki/Outer_product
• https://en.wikipedia.org/wiki/Tensor_product