Ch 2. Vector operations

• Prev: linalg-set-notation
• Next: matrix-operations

1. Vectors

Notation:

• A vector, $\text{v}$, is a list of numbers (scalars) arranged in a particular order.
• By convention, vectors are implicitly column vectors. So $\text{v}\in {\mathbb{R}}^n$; or implicitly $\text{v}\in {\mathbb{R}}^{n\times 1}$.
• To represent a column vector, $\text{v}$, as a row vector, take its transpose ${\text{v}}^{\mathrm{T}}$. Hence, ${\text{v}}^{\mathrm{T}}\in {\mathbb{R}}^{1\times n}$.
  • Transpose operation turns a column vector into a row vector vice versa. For matrices, it swaps rows and columns.
• The $i$‘th element of vector $\text{v}$ is denoted $v_i$.

Properties and geometric interpretation of a vector:

• The number of elements in a vector is called its dimension. This also tells us the vector space it belongs to.
  • For example, $\text{v}\in {\mathbb{R}}^3$ is an 3-dimensional vector.
• A vector can be thought of as an arrow in n-dimensional space.
• The magnitude of a vector is its length in n-dimensional space. (many distance formulas can be used to compute):
  • $L_2$ norm (Euclidean length): $\Vert \text{v}{\Vert }_2=\sqrt{{\Sigma }_i{w}_i^2}$ (physical length of a vector in n-dim space)
  • $L_1$ norm (Manhattan Distance): $\Vert \text{v}{\Vert }_1={\Sigma }_i|v_i|$
  • More generally,
    • The p-norm is $\Vert v{\Vert }_p=\sqrt[p]{({\sum }_i{v}_i^p)}$
    • And the $L_{\infty }$ norm, $\Vert \text{v}{\Vert }_{\infty }$, is the $p$-norm where $p=\infty$
  • Generally the $L_2$ norm is implied when referring to vector length.
    • Hence $\Vert \text{v}{\Vert }_2$ can be written more simply as $\Vert \text{v}\Vert$ or occasionally $|\text{v}|$ (though the latter is ambiguous with the absolute value of a scalar)

import numpy as np
 
# Create a row vector and a column vector, show their shapes
row_vec = np.array([[1, -5, 3, 2, 4]])
col_vec = np.array([[1], [2], [3], [4]])
 
# Comments show output if you don't use list of lists
print("row_vec =", row_vec[0], "\nrow_vec shape:", row_vec.shape)  # [ 1 -5  3  2  4] and (1, 5)
print("\ncol_vec =\n", col_vec, "\ncol_vec shape:", col_vec.shape)  # [1 2 3 4] and (4, 1)
 

row_vec = [ 1 -5  3  2  4] 
row_vec shape: (1, 5)
 
col_vec =
 [[1]
 [2]
 [3]
 [4]] 
col_vec shape: (4, 1)

# Transpose row_vec and calculate L1, L2, and L_inf norms
from numpy.linalg import norm
 
print("row_vec =", row_vec[0], "\nrow_vec shape:", row_vec.shape)
 
transposed_row_vec = row_vec.T  # now a column vector
print("\ntransposed_row_vec =", transposed_row_vec, "\ntransposed_row_vec shape:", transposed_row_vec.shape, "\n")
 
norm_1 = norm(transposed_row_vec, 1)
norm_2 = norm(transposed_row_vec, 2)  # <- L2 norm is default and most common
norm_inf = norm(transposed_row_vec, np.inf)
 
print("Measures of magnitude:")
print(f"L_1 norm is: {norm_1:.1f}")
print(f"L_2 norm is: {norm_2:.1f}", "(most common)")
print(f"L_inf norm is: {norm_inf:.1f}", "(absolute value of largest element)")  # NB: norm_inf = |largest element|
 

row_vec = [ 1 -5  3  2  4] 
row_vec shape: (1, 5)
 
transposed_row_vec = [[ 1]
 [-5]
 [ 3]
 [ 2]
 [ 4]] 
transposed_row_vec shape: (5, 1) 
 
Measures of magnitude:
L_1 norm is: 15.0
L_2 norm is: 7.4 (most common)
L_inf norm is: 5.0 (absolute value of largest element)

2. Vector addition, $\text{v}+\text{w}$

Elementwise addition; if vectors are of of same length (i.e. if $\text{v}$ and $\text{w}$ are both in ${\mathbb{R}}^n$) then:

• Addition of 2 vectors: $\text{u}=\text{v}+\text{w}$ is the vector with elements $u_i=v_i+w_i$

# Sum vectors v = [10, 9, 3] and w = [2, 5, 12]
v = np.array([[10, 9, 3]])
w = np.array([[2, 5, 12]])
 
print("Vector addition: \npy: v + w =", (v + w)[0])
 

Vector addition: 
py: v + w = [12 14 15]

3. Scalar times vector, $\alpha \text{v}$

To multiply a vector $\text{v}$, by a scalar $\alpha$ (a number in $\mathbb{R}$), do it “elementwise” or “pairwise”.

• Scalar multiplication of a vector: $\text{u}=\alpha \text{v}$ is the vector with elements $u_i=\alpha v_i$

# Multiply vector v = [10, 9, 3] by scalar alpha = 4
v = np.array([[10, 9, 3]])
alpha = 4
 
print("Scalar times vector: \npy: alpha * v =", (alpha * v)[0])
 

Scalar times vector: 
py: alpha * v = [40 36 12]

3.1. Linear Combinations

• A linear combination of set $S$ is defined as $\Sigma {\alpha }_i{s}_i$
  • Here ${\alpha }_i$ values are the coefficients of $s_i$ values
  • Example: Grocery bill total cost is a linear combination of items purchased:
    • $\sum c_i{n}_i$ ($c_i$ is item cost, $n_i$ is qty. purchased)

3.2. Linear Dependence and Independence

• A set is linearly INdependent if no object in the set can be written as a lin. combination of the other objects in the set.
  • Example: $\text{v}=[1,1,0],\text{w}=[1,0,0]$ and $\text{u}=[0,0,1]$ are linearly independent. Can you see why?

3.2.1. Geometric interpretation:

• If two vectors are linearly dependent, they lie on the same line/plane/hyperplane in n-dimensional space.
  • For a 2D vector space, if two vectors are linearly dependent, they lie on the same line (1D subspace).
  • For a 3D vector space, if two vectors are linearly dependent, they lie on the same plane (2D subspace).
  • And so on…

3.2.2. Example:

Below is an example of a linearly DEpendent set. $\text{x}$ is dependent on

# Writing the vector x = [-8, -1, 4] as a linear combination of 3 vectors, v, w, and u:
v = np.array([[0, 3, 2]])
w = np.array([[4, 1, 1]])
u = np.array([[0, -2, 0]])
 
x = 3 * v - 2 * w + 4 * u
print("x is a linear combination of v, w, and u. It is linearly dependent on them: \nx =", x[0])
 

x is a linear combination of v, w, and u. It is linearly dependent on them: 
x = [-8 -1  4]

4. Vector multiplication (4 approaches)

4.1. Vector dot (inner) product, $\text{v}\cdot \text{w}$

Geometric interpretation: A measure of how similarly directed two vectors are (think shadows, or projections)

• Definition: $\text{v}\cdot \text{w}={\Sigma }_{i=1}^n{v}_i{w}_i$
  • It’s the sum of elementwise products. For $\text{v},\text{w}\in {\mathbb{R}}^n$,
• Note also, since $\text{v},\text{w}\in {\mathbb{R}}^n$, transpose to make inner dimensions match. Dot product can hence be rewritten as:
  • $\text{v}\cdot \text{w}={\text{v}}^{\mathrm{T}}\text{w}$

Angle between vectors, $\theta$

• Think of dot product as “degree of alignment”: $\text{v}\cdot \text{w}=\Vert \text{v}\Vert \Vert \text{w}\Vert \cos \theta$
  • (1,1) and (2,2) are parallel; computing the angle gives $\theta =0$
  • (1,1) and (-1,1) are orthogonal (perpendicular) bc $\theta =\pi /2$ and $\text{v}\cdot \text{w}=0$ (no alignment)

• Extending on the dot product idea, $\text{v}\cdot \text{w}$, the angle between two vectors is $\theta$. It is as defined by the formula: $\theta =\arccos \left(\frac{\text{v}\cdot \text{w}}{\Vert \text{v}\Vert \Vert \text{w}\Vert }\right)$

Recap: Many ways to express a dot product (it’s commutative!):

$$\begin{array}{rl}\text{v}\cdot \text{w}={\Sigma }_{i=1}^n{v}_i{w}_i& ={\text{v}}^{\mathrm{T}}\text{w}=\Vert \text{v}\Vert \Vert \text{w}\Vert \cos \theta \\ & ={\text{w}}^{\mathrm{T}}\text{v}=\Vert \text{w}\Vert \Vert \text{v}\Vert \cos \theta =\text{w}\cdot \text{v}\end{array}$$

# Compute the dot (i.e. inner) product of vectors v = [10, 9, 3] and w = [2, 5, 12]
v = np.array([[10, 9, 3]])
w = np.array([[2, 5, 12]])
 
print("Vector dot product: v • w:")
print("py: np.dot(v, w.T) =", np.dot(v, w.T)[0][0])  # w.T to match inner dims, manually transpose w to column vector
 
# Compute the angle between vectors v and w
theta = np.arccos(np.dot(v, w.T) / (norm(v) * norm(w)))  # norm() default is L2
print("\nTheta =", theta[0][0])
 
# These methods produce the same result as np.dot(v, w)
print("\n----------------- Below methods produce same dot prod -----------------\n")
print("py: np.inner(v, w) =", np.inner(v, w)[0][0])  # same as dot, but more explicit. Can be used for higher dims.
print("py: np.matmul(v, w.T) =", np.matmul(v, w.T)[0][0])  # Assumes all inputs are row vectors
print("py: v @ w.T =", (v @ w.T)[0][0])  # @ operator is equivalent to np.matmul() for 2D arrays
 

Vector dot product: v • w:
py: np.dot(v, w.T) = 101
 
Theta = 0.979924710443726
 
----------------- Below methods produce same dot prod -----------------
 
py: np.inner(v, w) = 101
py: np.matmul(v, w.T) = 101
py: v @ w.T = 101

4.2. Vector outer product, $\text{v}\otimes \text{w}$

• Definition: $\text{v}\otimes \text{w}$ is the $m\times n$ matrix $\text{A}$ obtained by multiplying each element of $\text{v}$ by each element of $\text{w}$.

  • Unlike with the dot product, here the vectors $\text{v}$ and $\text{w}$ may have different lengths: $\text{v}\in {\mathbb{R}}^{m\times 1}$, $\text{w}\in {\mathbb{R}}^{n\times 1}$.
  • Elements of the resultant matrix are given by: $(\text{v}\otimes \text{w}{)}_{ij}=v_i{w}_j$
  • The entire matrix is represented as: $\text{v}\otimes \text{w}=\text{A}=\left[\begin{array}{cccc}{v}_1{w}_1& v_1{w}_2& \dots & v_1{w}_n\\ v_2{w}_1& v_2{w}_2& \dots & v_2{w}_n\\ ⋮& ⋮& \ddots & ⋮\\ v_m{w}_1& v_m{w}_2& \dots & v_m{w}_n\end{array}\right]$

• Note also, since $\text{v}$ and $\text{w}$ are column vectors ($\text{v}\in {\mathbb{R}}^{m\times 1}$, $\text{w}\in {\mathbb{R}}^{n\times 1}$), we can transpose ${\text{w}}^{\text{T}}$ (now a row vector, ${\text{w}}^{\mathrm{T}}\in {\mathbb{R}}^{1\times n}$).

  • $\text{v}\otimes \text{w}$ is hence equivalent to the matrix multiplication $\text{v}{\text{w}}^{\text{T}}$ (since this ensures the inner dimensions ($1$) to match). Example: $\text{v}\otimes \text{w}=\text{v}{\text{w}}^{\text{T}}=\left[\begin{array}{c}{v}_1\\ v_2\\ v_3\\ v_4\end{array}\right]\left[\begin{array}{ccc}{w}_1& w_2& w_3\end{array}\right]=\left[\begin{array}{ccc}{v}_1{w}_1& v_1{w}_2& v_1{w}_3\\ v_2{w}_1& v_2{w}_2& v_2{w}_3\\ v_3{w}_1& v_3{w}_2& v_3{w}_3\\ v_4{w}_1& v_4{w}_2& v_4{w}_3\end{array}\right]$

# Compute the outer product of same vectors v = [10, 9, 3] and w = [2, 5, 12]
v = np.array([[10, 9, 3]])
w = np.array([[2, 5, 12]])
# w = np.array([[2, 5, 12, 13]])  # <- uncomment to see outer product with different dimensions
 
print("Vectors:\nv =", v[0], "\nw =", w[0])
print("\nVector inner product, v • w\npy: np.inner(v, w) =", np.inner(v, w)[0][0])  # As previous code cell
print("\nVector outer product, v ⨂ w\nnp.outer(v, w) =\n", np.outer(v, w))  # Assumes all inputs column vectors
print("\nVector outer product flipped, w ⨂ v\nnp.outer(w, v) =\n", np.outer(w, v))  # Assumes all inputs column vectors
 

Vectors:
v = [10  9  3] 
w = [ 2  5 12]
 
Vector inner product, v • w
py: np.inner(v, w) = 101
 
Vector outer product, v ⨂ w
np.outer(v, w) =
 [[ 20  50 120]
 [ 18  45 108]
 [  6  15  36]]
 
Vector outer product flipped, w ⨂ v
np.outer(w, v) =
 [[ 20  18   6]
 [ 50  45  15]
 [120 108  36]]

4.3. Vector Hadamard (element-wise) product, $\text{v}\odot \text{w}$:

• Definition: Element-wise product, where the vectors must be of the same dimension
  • This method also works for matrices of the same dimension.
  • Output vector/matrix is of the same dimension as the operands.
• Notation: $\text{v}\odot \text{w}$ (sometimes $\text{v}\circ \text{w}$)
  • Elements of the resultant vector are given by: $(\text{v}\odot \text{w}{)}_i=(\text{v}{)}_i(\text{w}{)}_i$
• Example:

$\left[\begin{array}{ccc}2& 3& 1\end{array}\right]\odot \left[\begin{array}{ccc}3& 1& 4\end{array}\right]=\left[\begin{array}{ccc}2\times 3& 3\times 1& 1\times 4\end{array}\right]=\left[\begin{array}{ccc}6& 3& 4\end{array}\right]$

# Compute the Hadamard product of vectors v = [2, 3, 1] and w = [3, 1, 4]
v = np.array([[2, 3, 1]])
w = np.array([[3, 1, 4]])
 
print("Vector Hadamard product: v ⊙ w:\npy: np.multiply(v, w) =", np.multiply(v, w)[0])
 

Vector Hadamard product: v ⊙ w:
py: np.multiply(v, w) = [6 3 4]

4.4. Vector cross product, $\text{v}\times \text{w}$

Geometric interpretation: The cross product $\text{v}\times \text{w}$ is a vector perpendicular to both $\text{v}$ and $\text{w}$, whose length equals the area enclosed by the parallelogram created by the two vectors

• Cross product definition: $\text{v}\times \text{w}=\Vert \text{v}\Vert \Vert \text{w}\Vert \sin (\theta )\text{n}$
• Where:
  • $\theta$ can by computed via dot product
  • $\text{n}$ is a unit vector (i.e. length $1$) perpendicular to both $\text{v}$ and $\text{w}$.

# Compute the cross product of v = [0,2,0] and w = [3,0,0]
v = np.array([[0, 2, 0]])
w = np.array([[3, 0, 0]])
 
print("Vector cross product: v ⨉ w\nnp.cross(v, w) =", np.cross(v, w)[0])
 

Vector cross product: v ⨉ w
np.cross(v, w) = [ 0  0 -6]