PolarSPARC

Introduction to Linear Algebra - Part 1


Bhaskar S 01/29/2022


Basic Definitions

The following are some of the basic definitions that will be referred in to following paragraphs:

Name Description
Scalar is just a number (or a constant) and is represented using the lowercase greek alphabets - α,β,γ,λ, etc.
Vector a quantity that has both magnitude and direction, and is represented as a lowercase letter with an arrow on the top - a
Matrix (plural Matrices) a rectangular table of mathematical elements that is arranged in rows and columns, and is represented with an appercase letter.
Example: [358249]

Linear Algebra is the branch of mathematics that deals with Vectors and Matrices.

Vectors

The following are some of the characteristics of Vectors:

Dot Product of Vectors

The Dot (also known as the Scalar or Inner) product of two vectors (of same dimensionality) is a single scalar. To perform the dot product of the two vectors, multiply the corresponding elements of the two vectors and then sum each of the individual products. In other words, if a and b are the two vectors, then aT.b=i=1naibi.

Let us look at an example for the dot product.

Example-1 Find the dot product of the vectors [216] and [321]

The dot product is computed using the formula aT.b=i=1naibi.

That is, (23)+(12)+(61)=62+6=10


The following are some of the properties of Dot Product (of vectors):

Magnitude of a Vector

The Magnitude (also known as the Length or Norm) of a vector a is represented as a and equals aTa.

We will use a geometric perspective to find the magnitude of a vector for a better intuition. A vector a in a two dimensional space (since it is easy to visualize) can be represented using a line from the origin (0,0) to some coordinate point (x,y).

The following graph is the illustration of the vector:

Vector Length
Fig.1

Using the Pythagorean Theorem, we can infer the length of a [from (0,0) to (x,y)] as x2+y2=aTa.

Extending this concept to a vector a of dimensionality N, a=i=1Nai2

Geometric Perspective of the Dot Product

From a geometric perspective, the dot product of two vectors a and b in a two dimensional space is the cosine of the angle between the two vectors multiplied with the lengths of the two vectors.

In other words, aTb=i=1naibi=abcosθ, where θ is the angle between the two vectors a and b.

Rearranging the equation, we get: cosθ=aTbab.

Or, θ=arccos(aTbab), where arccos=cos1.

The following graph is the illustration of the two vectors:

Vector Dot Product
Fig.2

From the triangle above, using the Pythagorean Theorem, we can infer the following:

c2=p2+q2 ... (1)

a2=p2+(bq)2 ... (2)

Using Trigonometry, we can infer the following:

sinθ=pa OR p=asinθ ... (3)

cosθ=bqa OR q=bacosθ ... (4)

Expanding the equation (2) from above, p2+(b2+q22bq)=a2

That is, (p2+q2)+b22bq=a2

Replacing the first term with c2 from equation (1) above, we get c2+b22bq=a2

Replacing q from equation (4) above, we get c2+b22b(bacosθ)=a2

Simplifying, we get c2=a2+b22abcosθ=a2+b22abcosθ

This is the equation for the Law of Cosines.

NOTICE that when θ=90, the two vectors a and b are said to be orthogonal to each other and cosθ=0. The equation for the Law of Cosines morphs into the equation for the Pythagorean Theorem.

From geometry, we can infer c=ab

In other words, c=ab

We know a=aTa. Therefore, rewriting the above equation as:

c=(ab)T(ab)=aTa2aTb+bTb=a2+b22aTb

Substituting in the equation for the law of cosines, we get:

a2+b22aTb=a2+b22abcosθ

Simplifying the equation, we get: aTb=abcosθ, thus proving the geometric interpretation of the vector dot product.


*** ATTENTION ***

  For orthogonal vectors a and b, where θ=90, cos(θ)=cos(90)=0 and therefore aTb=0


Outer Product of Vectors

The Outer product of two vectors (of same or different dimensionality) results in a matrix. In other words, if a has a dimensionality of N and b has a dimensionality of M, then the Outer Product of the two vectors is a matrix of N x M elements.

An example will make it clear

Let a=[4132]

Let b=[abc]

abT=[4132][abc]=[4.a4.b4.c1.a1.b1.c3.a3.b3.c2.a2.b2.c]=[4a4b4cabc3a3b3c2a2b2c]

For each element ai in the column vector a, scale the row vector b using ai.b to form the new row of the output matrix.

Unit Vector

A Unit Vector is a vector whose magnitude (or length) equals one (1), that is, a=1.

Given a vector a that is of a certain magnitude and in a certain direction, the idea is to find a scalar μ such that the magnitude of the scaled vector μa=1. A unit vector of a vector a is represented as a^ (pronouned a-HAT).

Given μa=1, we can simplify it to derive μ=1a.

Therefore, a Unit Vector for a is given as a^=1a a=1i=1Nai a.

Let us look at an example now.

Example-2 Find the Unit Vector for the vector [34]

We know a Unit Vector of a given vector a is a^=1a a.

We also know a=i=1Nai =32+42=25=5.

RESULT: the Unit Vector for [34] equals 15 [34]=[3/54/5]


Direction of a Vector

The Direction is the second aspect of a vector a (first being the Magnitude) that is represented as another vector a^ and equals aa.

We will use a geometric perspective to find the direction of a vector for a better intuition. A vector a in a two dimensional space (since it is easy to visualize) can be represented using a line from the origin (0,0) to some coordinate point (x,y) and θ is the angle between the horizontal x-axis and the vector a.

The following graph is the illustration of the vector:

Vector Direction
Fig.3

From Trigonometry, we know cos(θ)=adjacenthypotenuse.

That is, cos(θ)=xa.

Similarly, cos(90θ)=ya.

Therefore, a^= ( xa , ya ).

Extending this concept to a vector a of dimensionality N, a^= ( a1a , a2a ,..., aNa ).

Field

As indicated above, a Dimension is the number of elements in a vector. To get an intuitive understanding, one can look at it from a geometric perspective - a one-dimensional vector represents a line, a 2-dimensional vector represents a plane, a 3-dimensional vector represents a 3-dimensional space (also known as the Euclidean Space), and so on.

A Field is a set of numbers in a particular Dimension, on which, the four basic mathematical operations, namely, addition, subtraction, multiplication, and division are valid. In mathematics, a Field is reprseneted as a hollow capital letter, such as R, which represents a Field of real numbers, etc.

One can associate Dimensionality with a Field by raising the Field letter with the Dimension. For example, the Field R4 represents a Field of all four-dimensional vectors of the form [abcd]. In general, an N-dimensional Field of real number vectors is represented as RN.

Subspaces

A Subspace is the set of all vectors that can be created by all possible linear combinations of vector-scalar multiplication for a given set of vectors in RN and all scalars in R. Subspaces are often indicated using italicized upper-case letters, such as, V.

In other words, a vector subspace must satisfy the following criteria:

The simplest linear combination is a scalar multiplication operation (scaling) of a vector. That is, for a given vector a, one can create the other vectors in the same Subspace by performing a scalar multiplication λa, where the scalar λR. The set of new vectors created from λa is said to be in the same Subspace as the original vector a. Another linear combination could be the addition operation on two or more vectors (from the same Subspace) such as λa+μb or λaμb.

In mathematical terms, if a,bV and λ,μR, then λa+μbV, where V is a Vector Subspace.

Linear Dependence

A set of vectors are said to be Linearly Dependent if some scalar combination of the vectors can form the zero vector (a vector with all zero elements).

Mathematically, λ1a1+λ2a2+...+λnan=0n, where λ1,λ2,..,λnR, where atleast one λn0.

The following set of vectors from the 3-dimensional space are linearly dependant:

a=[072]

b=[126]

c=[2176]

We can show (3)a+2b+c=[000]

Basis Vector

A Basis is a set of vectors that are linearly independent of each other and can be linearly combined to generate other vectors in the vector space they belong to.

The unit vectors [10] and [01] from the 2-dimensional vector space are referred to as the Standard Orthogonal basis vectors.



© PolarSPARC