PolarSPARC

Introduction to Linear Algebra - Part 1


Bhaskar S 01/29/2022

Basic Definitions

The following are some of the basic definitions that will be referred in to following paragraphs:

Name Description
Scalar is just a number (or a constant) and is represented using the lowercase greek alphabets - $\alpha, \beta, \gamma, \lambda$, etc.
Vector a quantity that has both magnitude and direction, and is represented as a lowercase letter with an arrow on the top - $\vec{a}$
Matrix (plural Matrices) a rectangular table of mathematical elements that is arranged in rows and columns, and is represented with an appercase letter.
Example: $\begin{bmatrix} 3 & 5 & 8 \\ 2 & 4 & 9 \end{bmatrix}$

Linear Algebra is the branch of mathematics that deals with Vectors and Matrices.

Vectors

The following are some of the characteristics of Vectors:

Dot Product of Vectors

The Dot (also known as the Scalar or Inner) product of two vectors (of same dimensionality) is a single scalar. To perform the dot product of the two vectors, multiply the corresponding elements of the two vectors and then sum each of the individual products. In other words, if $\vec{a}$ and $\vec{b}$ are the two vectors, then $\vec{a}^T.\vec{b} = \sum\limits_{i=1}^n\vec{a_i}\vec{b_i}$.

Let us look at an example for the dot product.

Example-1 Find the dot product of the vectors $\begin{bmatrix} 2 \\ -1 \\ 6 \end{bmatrix}$ and $\begin{bmatrix} 3 \\ 2 \\ 1 \end{bmatrix}$

The dot product is computed using the formula $\vec{a}^T.\vec{b} = \sum\limits_{i=1}^n\vec{a_i}\vec{b_i}$.

That is, $(2 * 3) + (-1 * 2) + (6 * 1) = 6 -2 + 6 = 10$

The following are some of the properties of Dot Product (of vectors):

Magnitude of a Vector

The Magnitude (also known as the Length or Norm) of a vector $\vec{a}$ is represented as $\lVert\vec{a}\rVert$ and equals $\sqrt{\vec{a}^T \vec{a}}$.

We will use a geometric perspective to find the magnitude of a vector for a better intuition. A vector $\vec{a}$ in a two dimensional space (since it is easy to visualize) can be represented using a line from the origin $(0, 0)$ to some coordinate point $(x, y)$.

The following graph is the illustration of the vector:

Vector Length
Fig.1

Using the Pythagorean Theorem, we can infer the length of $\vec{a}$ [from $(0, 0)$ to $(x, y)$] as $\sqrt{x^2 + y^2} = \vec{a}^T\vec{a}$.

Extending this concept to a vector $\vec{a}$ of dimensionality N, $\lVert\vec{a}\rVert = \sqrt{\sum\limits_{i=1}^N\vec{a_i}^2}$

Geometric Perspective of the Dot Product

From a geometric perspective, the dot product of two vectors $\vec{a}$ and $\vec{b}$ in a two dimensional space is the cosine of the angle between the two vectors multiplied with the lengths of the two vectors.

In other words, $a^Tb = \sum\limits_{i=1}^n\vec{a_i}\vec{b_i} = \lVert \vec{a} \rVert \lVert \vec{b} \rVert \cos \theta$, where $\theta$ is the angle between the two vectors $\vec{a}$ and $\vec{b}$.

Rearranging the equation, we get: $\cos \theta = \Large{\frac{a^Tb}{\lVert \vec{a} \rVert \lVert \vec{b} \rVert}}$.

Or, $\theta = \arccos\Large{(\frac{a^Tb}{\lVert \vec{a} \rVert \lVert \vec{b} \rVert})}$, where $\arccos = \cos^{-1}$.

The following graph is the illustration of the two vectors:

Vector Dot Product
Fig.2

From the triangle above, using the Pythagorean Theorem, we can infer the following:

$c^2 = p^2 + q^2$ ... (1)

$a^2 = p^2 + (b-q)^2$ ... (2)

Using Trigonometry, we can infer the following:

$\sin \theta = \Large{\frac{p}{a}}$ OR $p = a \sin \theta$ ... (3)

$\cos \theta = \Large{\frac{b-q}{a}}$ OR $q = b - a \cos \theta$ ... (4)

Expanding the equation (2) from above, $p^2 + (b^2 + q^2 - 2bq) = a^2$

That is, $(p^2 + q^2) + b^2 -2bq = a^2$

Replacing the first term with $c^2$ from equation (1) above, we get $c^2 + b^2 -2bq = a^2$

Replacing q from equation (4) above, we get $c^2 + b^2 -2b(b - a \cos \theta) = a^2$

Simplifying, we get $c^2 = a^2 + b^2 - 2ab \cos \theta = {\lVert \vec{a} \rVert}^2 + {\lVert \vec{b} \rVert}^2 - 2\lVert \vec{a} \rVert \lVert \vec{b} \rVert \cos \theta$

This is the equation for the Law of Cosines.

NOTICE that when $\theta = 90^{\circ}$, the two vectors $\vec{a}$ and $\vec{b}$ are said to be orthogonal to each other and $\cos \theta = 0$. The equation for the Law of Cosines morphs into the equation for the Pythagorean Theorem.

From geometry, we can infer $\vec{c} = \vec{a} - \vec{b}$

In other words, $\lVert \vec{c} \rVert = \lVert \vec{a} \rVert - \lVert \vec{b} \rVert$

We know $\lVert \vec{a} \rVert = a^Ta$. Therefore, rewriting the above equation as:

$\lVert \vec{c} \rVert = (a - b)^T(a - b) = a^Ta - 2a^Tb + b^Tb = {\lVert \vec{a} \rVert}^2 + {\lVert \vec{b} \rVert}^2 - 2a^Tb$

Substituting in the equation for the law of cosines, we get:

${\lVert \vec{a} \rVert}^2 + {\lVert \vec{b} \rVert}^2 - 2a^Tb = {\lVert \vec{a} \rVert}^2 + {\lVert \vec{b} \rVert}^2 - 2ab \cos \theta$

Simplifying the equation, we get: $a^Tb = \lVert \vec{a} \rVert \lVert \vec{b} \rVert \cos \theta$, thus proving the geometric interpretation of the vector dot product.


*** ATTENTION ***

  For orthogonal vectors $\vec{a}$ and $\vec{b}$, where $\theta = 90^{\circ}$, $\cos(\theta) = \cos(90) = 0$ and therefore $a^Tb = 0$


Outer Product of Vectors

The Outer product of two vectors (of same or different dimensionality) results in a matrix. In other words, if $\vec{a}$ has a dimensionality of N and $\vec{b}$ has a dimensionality of M, then the Outer Product of the two vectors is a matrix of N x M elements.

An example will make it clear

Let $\vec{a} = \begin{bmatrix} 4 \\ 1 \\ 3 \\ 2 \end{bmatrix}$

Let $\vec{b} = \begin{bmatrix} a & b & c \end{bmatrix}$

$\vec{a}\vec{b}^T = \begin{bmatrix} 4 \\ 1 \\ 3 \\ 2 \end{bmatrix} \begin{bmatrix} a & b & c \end{bmatrix} = \begin{bmatrix} 4.a & 4.b & 4.c \\ 1.a & 1.b & 1.c \\ 3.a & 3.b & 3.c \\ 2.a & 2.b & 2.c \end{bmatrix} = \begin{bmatrix} 4a & 4b & 4c \\ a & b & c \\ 3a & 3b & 3c \\ 2a & 2b & 2c \end{bmatrix}$

For each element $a_i$ in the column vector $\vec{a}$, scale the row vector $\vec{b}$ using $a_i.\vec{b}$ to form the new row of the output matrix.

Unit Vector

A Unit Vector is a vector whose magnitude (or length) equals one (1), that is, $\lVert \vec{a} \rVert = 1$.

Given a vector $\vec{a}$ that is of a certain magnitude and in a certain direction, the idea is to find a scalar $\mu$ such that the magnitude of the scaled vector $\lVert \mu \vec{a} \rVert = 1$. A unit vector of a vector $\vec{a}$ is represented as $\hat{a}$ (pronouned a-HAT).

Given $\lVert \mu \vec{a} \rVert = 1$, we can simplify it to derive $\mu = \Large{\frac{1}{\lVert \vec{a} \rVert}}$.

Therefore, a Unit Vector for $\vec{a}$ is given as $\hat{a} = \Large{\frac{1}{\lVert \vec{a} \rVert}}$ $\vec{a} = \Large{\frac{1}{\sqrt{\sum\limits_{i=1}^N\vec{a_i}}}}$ $\vec{a}$.

Let us look at an example now.

Example-2 Find the Unit Vector for the vector $\begin{bmatrix} 3 \\ 4 \end{bmatrix}$

We know a Unit Vector of a given vector $\vec{a}$ is $\hat{a} = \Large{\frac{1}{\lVert \vec{a} \rVert}}$ $\vec{a}$.

We also know $\lVert\vec{a}\rVert = \sqrt{\sum\limits_{i=1}^N\vec{a_i}}$ $= \sqrt{3^2 + 4^2} = \sqrt{25} = 5$.

RESULT: the Unit Vector for $\begin{bmatrix} 3 \\ 4 \end{bmatrix}$ equals $\Large{\frac{1}{5}}$ $\begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 3/5 \\ 4/5 \end{bmatrix}$

Direction of a Vector

The Direction is the second aspect of a vector $\vec{a}$ (first being the Magnitude) that is represented as another vector $\hat{a}$ and equals $\Large{\frac{\vec{a}}{\lVert\vec{a}\rVert}}$.

We will use a geometric perspective to find the direction of a vector for a better intuition. A vector $\vec{a}$ in a two dimensional space (since it is easy to visualize) can be represented using a line from the origin $(0, 0)$ to some coordinate point $(x, y)$ and $\theta$ is the angle between the horizontal x-axis and the vector $\vec{a}$.

The following graph is the illustration of the vector:

Vector Direction
Fig.3

From Trigonometry, we know $cos(\theta) = \Large{\frac{adjacent}{hypotenuse}}$.

That is, $cos(\theta) = \Large{\frac{x}{\lVert\vec{a}\rVert}}$.

Similarly, $cos(90-\theta) = \Large{\frac{y}{\lVert\vec{a}\rVert}}$.

Therefore, $\hat{a} =$ $($ $\Large{\frac{x}{\lVert\vec{a}\rVert}}$ $,$ $\Large{\frac{y}{\lVert\vec{a}\rVert}}$ $)$.

Extending this concept to a vector $\vec{a}$ of dimensionality N, $\hat{a} = $ $($ $\Large{\frac{a_1}{\lVert\vec{a}\rVert}}$ $,$ $\Large{\frac{a_2}{\lVert\vec{a}\rVert}}$ $, ..., $ $\Large{\frac{a_N}{\lVert\vec{a}\rVert}}$ $)$.

Field

As indicated above, a Dimension is the number of elements in a vector. To get an intuitive understanding, one can look at it from a geometric perspective - a one-dimensional vector represents a line, a 2-dimensional vector represents a plane, a 3-dimensional vector represents a 3-dimensional space (also known as the Euclidean Space), and so on.

A Field is a set of numbers in a particular Dimension, on which, the four basic mathematical operations, namely, addition, subtraction, multiplication, and division are valid. In mathematics, a Field is reprseneted as a hollow capital letter, such as $\mathbb{R}$, which represents a Field of real numbers, etc.

One can associate Dimensionality with a Field by raising the Field letter with the Dimension. For example, the Field $\mathbb{R}^4$ represents a Field of all four-dimensional vectors of the form $\begin{bmatrix} a & b & c &d \end{bmatrix}$. In general, an N-dimensional Field of real number vectors is represented as $\mathbb{R}^N$.

Subspaces

A Subspace is the set of all vectors that can be created by all possible linear combinations of vector-scalar multiplication for a given set of vectors in $\mathbb{R}^N$ and all scalars in $\mathbb{R}$. Subspaces are often indicated using italicized upper-case letters, such as, $\textit{V}$.

In other words, a vector subspace must satisfy the following criteria:

The simplest linear combination is a scalar multiplication operation (scaling) of a vector. That is, for a given vector $\vec{a}$, one can create the other vectors in the same Subspace by performing a scalar multiplication $\lambda \vec{a}$, where the scalar $\lambda \in \mathbb{R}$. The set of new vectors created from $\lambda \vec{a}$ is said to be in the same Subspace as the original vector $\vec{a}$. Another linear combination could be the addition operation on two or more vectors (from the same Subspace) such as $\lambda\vec{a} + \mu\vec{b}$ or $\lambda\vec{a} - \mu\vec{b}$.

In mathematical terms, if $\vec{a}, \vec{b} \in \textit{V}$ and $\lambda, \mu \in \mathbb{R}$, then $\lambda\vec{a} + \mu\vec{b} \in V$, where $\textit{V}$ is a Vector Subspace.

Linear Dependence

A set of vectors are said to be Linearly Dependent if some scalar combination of the vectors can form the zero vector (a vector with all zero elements).

Mathematically, $\lambda_1\vec{a_1} + \lambda_2\vec{a_2} + ... + \lambda_n\vec{a_n} = \textbf{0}_n$, where $\lambda_1, \lambda_2, .., \lambda_n \in \mathbb{R}$, where atleast one $\lambda_n \ne 0$.

The following set of vectors from the 3-dimensional space are linearly dependant:

$\vec{a} = \begin{bmatrix} 0 \\ 7 \\ 2 \end{bmatrix}$

$\vec{b} = \begin{bmatrix} 1 \\ 2 \\ 6 \end{bmatrix}$

$\vec{c} = \begin{bmatrix} -2 \\ 17 \\ -6 \end{bmatrix}$

We can show $(-3)\vec{a} + 2\vec{b} + \vec{c} = \begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix}$

Basis Vector

A Basis is a set of vectors that are linearly independent of each other and can be linearly combined to generate other vectors in the vector space they belong to.

The unit vectors $\begin{bmatrix} 1 \\ 0 \end{bmatrix}$ and $\begin{bmatrix} 0 \\ 1 \end{bmatrix}$ from the 2-dimensional vector space are referred to as the Standard Orthogonal basis vectors.


© PolarSPARC