PolarSPARC |
Introduction to Linear Algebra - Part 2
Bhaskar S | 02/06/2022 |
In Part 1 of this series, we covered the topic of Vectors. In this part, we will cover the topic on Matrices.
Matrices
The following are some of the characteristics of Matrices:
is represented using a bold-face capital letter, such as, $\textbf{A}$
is a two dimensional table of elements (numbers or variables) with rows and columns
individual elements are represented using a lowercase letter with a subscript, such as, $a_{i,j}$, where the subscript i represents the row and the subscript j represents the column
the size is represented as M x N, where M is the number of rows and N is the number of columns
The following is an example of a 3 x 5 matrix:
$\begin{bmatrix} 3 & 9 & -1 & 2 & -5 \\ 8 & 2 & 0 & -7 & 9 \\ -4 & 7 & 9 & 2 & -3 \end{bmatrix}$
Matrix Transpose
The transpose of a matrix $\textbf{A}$, represented as $\textbf{A}^T$, is a new matrix where the rows and columns get swapped. In other words, the rows become the columns in the new matrix. Mathematically, every element $x_{i,j}$ in the given matrix $\textbf{A}$ becomes the element $x_{j,i}$ in the transposed matrix $\textbf{A}^T$.
The following is a 3 x 4 matrix $\textbf{A}$:
$\textbf{A} = \begin{bmatrix} 3 & 9 & -1 & 2 \\ 8 & 2 & 0 & -7 \\ -4 & 7 & 9 & 2 \end{bmatrix}$
The following is the transpose of the matrix $\textbf{A}$, which results in a 4 x 3 matrix $\textbf{A}^T$:
$\textbf{A}^T = \begin{bmatrix} 3 & 8 & -4 \\ 9 & 2 & 7 \\ -1 & 0 & 9 \\ 2 & -7 & 2 \end{bmatrix}$
Zero Matrix
A matrix that is represented as $\textbf{O}$ in which all the elements are zeros.
The following is an example of a 3 x 3 zero matrix:
$\begin{bmatrix} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix}$
Square Matrix
When the number of rows and columns are equal, it is a Square matrix.
The following is an example of a 3 x 3 square matrix:
$\begin{bmatrix} 3 & 9 & -1 \\ 8 & 2 & 0 \\ -4 & 7 & 9 \end{bmatrix}$
Symmetric Matrix
A sqaure matrix such that the matrix $\textbf{A}$ equals its transpose $\textbf{A}^T$.
The following is an example of a 3 x 3 symmetric matrix:
$\begin{bmatrix} 3 & 2 & 4 \\ 2 & 5 & 6 \\ 4 & 6 & 7 \end{bmatrix}$
Identity Matrix
A square matrix (of size N x N) that is represented as $\mathbf{I_N}$ in which all the diagonal elements are ones, while all the others elements are zeros.
The following is an example of a 3 x 3 identity matrix:
$\begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}$
Diagonal Matrix
A matrix in which all the elements outside the elements along the diagonal are zeros.
The following is an example of a 3 x 3 diagonal matrix:
$\begin{bmatrix} 1 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 0 & 3 \end{bmatrix}$
The following is an example of a 4 x 3 diagonal matrix:
$\begin{bmatrix} 1 & 0 & 0 \\ 0 & 3 & 0 \\ 0 & 0 & -5 \\ 0 & 0 & 0 \end{bmatrix}$
Equality of Matrices
For two matrices $\textbf{A}$ and $\textbf{B}$ to be equal, they first need to be of the same size M x N, and second each of the corresponding elements in the two matrices must be equal, that is, $a_{i,j} = b_{i,j}$.
The following is the 3 x 4 matrix $\textbf{A}$:
$\textbf{A} = \begin{bmatrix} 3 & 9 & -1 & 2 \\ 8 & 2 & 0 & -7 \\ -4 & 7 & 9 & 2 \end{bmatrix}$
The following is the 3 x 4 matrix $\textbf{B}$:
$\textbf{B} = \begin{bmatrix} 3 & 9 & -1 & 2 \\ 8 & 2 & 0 & -7 \\ -4 & 7 & 9 & 2 \end{bmatrix}$
The following is the 3 x 4 matrix $\textbf{C}$:
$\textbf{C} = \begin{bmatrix} 3 & 9 & -1 & 2 \\ 8 & 2 & 0 & -7 \\ -4 & \textcolor{red}{0} & 9 & 2 \end{bmatrix}$
The two matrices $\textbf{A}$ and $\textbf{B}$ are EQUAL, while the matrices $\textbf{B}$ and $\textbf{C}$ are NOT equal (because of the element in $\textcolor{red}{RED}$ in matrix $\textbf{C}$).
Matrix Addition and Subtraction
To add or subtract two matrices $\textbf{A}$ and $\textbf{B}$ of the same size M x N, just add or subtract each of the corresponding elements in the two matrices, that is, $a_{i,j} \pm b_{i,j}$.
The following is the 3 x 4 matrix $\textbf{A}$:
$\textbf{A} = \begin{bmatrix} 3 & 9 & -1 & 2 \\ 8 & 2 & 0 & -7 \\ -4 & 7 & 9 & 2 \end{bmatrix}$
The following is the 3 x 4 matrix $\textbf{B}$:
$\textbf{B} = \begin{bmatrix} -1 & 0 & 2 & 4 \\ -3 & 1 & 0 & 6 \\ 7 & 0 & -3 & 4 \end{bmatrix}$
The following is the result of adding matrices $\textbf{A}$ and $\textbf{B}$:
$\textbf{A} + \textbf{B} = \begin{bmatrix} 2 & 9 & 1 & 6 \\ 5 & 3 & 0 & -1 \\ 3 & 7 & 6 & 6 \end{bmatrix}$
The following is the result of subtracting matrices $\textbf{A}$ and $\textbf{B}$:
$\textbf{A} - \textbf{B} = \begin{bmatrix} 4 & 9 & -3 & -2 \\ 11 & 1 & 0 & -13 \\ -11 & 7 & 12 & -2 \end{bmatrix}$
Matrix Scalar Multiplication
To multiple a scalar $\lambda$ with a matrix $\textbf{A}$, just multiple the scalar with each of the elements in the matrix, that is, $\lambda.a_{i,j}$.
The following is the 3 x 4 matrix $\textbf{A}$:
$\textbf{A} = \begin{bmatrix} 3 & 9 & -1 & 2 \\ 8 & 2 & 0 & -7 \\ -4 & 7 & 9 & 2 \end{bmatrix}$
The following is the result of multiplying $\lambda = 2$ with the matrix $\textbf{A}$:
$2 . \textbf{A} = \begin{bmatrix} 6 & 18 & -2 & 4 \\ 16 & 4 & 0 & -14 \\ -8 & 14 & 18 & 4 \end{bmatrix}$
Matrix Multiplication
To multiply two matrices $\textbf{A}$ and $\textbf{B}$, the first matrix $\textbf{A}$ must be of size M x N and the second matrix $\textbf{B}$ must be of size N x P. In other words, the number of columns in the first matrix $\textbf{A}$ MUST match the number of rows in the second matrix $\textbf{B}$. Each element in the new matrix is a linear combination represented as $\sum\limits_{k=1}^N a_{i,k}.b_{k,j}$, where $i$ represents a row and $j$ represents a column. The resulting new matrix will be of size M x P.
The following is the 3 x 2 matrix $\textbf{A}$:
$\textbf{A} = \begin{bmatrix} a & b \\ c & d \\ e & f \end{bmatrix}$
The following is the 2 x 2 matrix $\textbf{B}$:
$\textbf{B} = \begin{bmatrix} p & q \\ r & s \end{bmatrix}$
The following is the result of multiplying the two matrices $\textbf{A}$ and $\textbf{B}$:
$\textbf{A}\textbf{B} = \begin{bmatrix} a.p+b.r & a.q+b.s \\ c.p+d.r & c.q+d.s \\ e.p+f.r & e.q+f.s \end{bmatrix}$
The following is a pictorial illustration of the matrix multiplication:
Note that the order of matrices involved in the multiplication matter, meaning, $\textbf{A}\textbf{B} \ne \textbf{B}\textbf{A}$.
Let us look at an example for the matrix multiplication.
Example-1 | Multiple the following two matrices: $\textbf{A} = \begin{bmatrix} 3 & 9 \\ -1 & 2 \\ 8 & 2 \\ 0 & -7 \end{bmatrix}$ and $\textbf{B} = \begin{bmatrix} 2 & 3 \\ -1 & 4 \end{bmatrix}$ |
---|---|
Two multiply the two matrices $\textbf{A}$ and $\textbf{B}$, each element in the new matrix is a linear combination represented as $\sum\limits_{k=1}^N a_{i,k}.b_{k,j}$, where $i$ represents a row and $j$ represents a column. That is, $\textbf{A}\textbf{B} = \begin{bmatrix} 3.2+9.(-1) & 3.3+9.4 \\ (-1).2+2.(-1) & (-1).3+2.4 \\ 8.2+2.(-1) & 8.3+2.4 \\ 0.2+(-7).(-1) & 0.3+(-7).4 \end{bmatrix} = \begin{bmatrix} -3 & 45 \\ -4 & 5 \\ 14 & 32 \\ 7 & -28 \end{bmatrix}$ |
Let us look at another example for the matrix multiplication.
Example-2 | Multiple the following two matrices: $\textbf{A} = \begin{bmatrix} 2 & 1 & 3 \\ 5 & 4 & 6 \end{bmatrix}$ and $\textbf{B} = \begin{bmatrix} 2 \\ 3 \\ 4 \end{bmatrix}$ |
---|---|
$\textbf{A}\textbf{B} = \begin{bmatrix} 2.2+1.3+3.4 \\ 5.2+4.3+6.4 \end{bmatrix} = \begin{bmatrix} 19 \\ 46 \end{bmatrix}$ |
Let us look at another interesting example of matrix multiplication involving a diagonal matrix.
Example-3 | Multiple the following two matrices: $\textbf{A} = \begin{bmatrix} a & b & c \\ d & e & f \\ g & h & i \end{bmatrix}$ and $\textbf{B} = \begin{bmatrix} 2 & 0 & 0 \\ 0 & 4 & 0 \\ 0 & 0 & 6 \end{bmatrix}$ |
---|---|
$\textbf{A}\textbf{B} = \begin{bmatrix} 2.a+0.b+0.c & 0.a+4.b+0.c & 0.a+0.b+6.c \\ 2.d+0.e+0.f & 0.d+4.e+0.f & 0.d+0.e+6.f \\ 2.g+0.h+0.i & 0.g+4.h+0.i & 0.g+0.h+6.i \end{bmatrix} = \begin{bmatrix} 2a & 4b & 6c \\ 2d & 4e & 6f \\ 2g & 4h & 6i \end{bmatrix}$ Notice how each column in the matrix $\textbf{A}$ is scaled by the corresponding column element along the diagonal of the matrix $\textbf{B}$. |
Properties of Matrices
The following are some of the properties of matrix addition and matrix scalar multiplication:
Matrix addition of the two matrices $\textbf{A}$ and $\textbf{B}$ of the same size is Commutative. That is: $\textbf{A} + \textbf{B} = \textbf{B} + \textbf{A}$
Matrix addition of the matrix $\textbf{A}$ with the zero matrix $\textbf{O}$ of the same size is matrix $\textbf{A}$. That is: $\textbf{A} + \textbf{O} = \textbf{A}$
Matrix addition of the matrix $\textbf{A}$ with the negative matrix $\textbf{A}$ is the zero matrix $\textbf{O}$. That is: $\textbf{A} + (-\textbf{A}) = \textbf{O}$
Matrix addition of the three matrices $\textbf{A}$, $\textbf{B}$, and $\textbf{C}$ of the same size is Associative. That is: $(\textbf{A} + \textbf{B}) + \textbf{C} = \textbf{A} + (\textbf{B} + \textbf{C})$
Matrix scalar multiplication on the addition of two matrices $\textbf{A}$ and $\textbf{B}$ of the same size is Distributive. That is: $\lambda.(\textbf{A} + \textbf{B}) = \lambda.\textbf{A} + \lambda.\textbf{B}$
Matrix transpose on the addition of two matrices $\textbf{A}$ and $\textbf{B}$ of the same size is equal to the sum of matrix transpose of $\textbf{A}$ and matrix transpose of $\textbf{B}$. That is: $(\textbf{A} + \textbf{B})^T = \textbf{A}^T + \textbf{B}^T$
Matrix multiplication of $\textbf{A}$ with the addition of two other matrices $\textbf{B}$ and $\textbf{C}$ is Distributive. That is: $\textbf{A}(\textbf{B} + \textbf{C}) = \textbf{A}\textbf{B} + \textbf{A}\textbf{C}$
Given a matrix $\textbf{A}$ of size M x N, the matrix multiplication with an identity matrix $\mathbf{I_N}$ (of size N x N) results in the same matrix $\textbf{A}$. In other words, $\textbf{A}\mathbf{I_N} = \textbf{A}$. Similarly, the matrix multiplication of an identity matrix $\mathbf{I_M}$ (of size M x M) with the matrix $\textbf{A}$ of size M x N results in the same matrix $\textbf{A}$. In other words, $\mathbf{I_M}\textbf{A} = \textbf{A}$
System of Linear Equations
A System of Linear Equations is a set of M linear equations that consist of the same set of N variables.
The following is a generic system of linear:
$\begin{align*} & a_{11}.x_1 + a_{12}.x_2 + ... + a_{1N}.x_N = b_1 \\ & a_{21}.x_1 + a_{22}.x_2 + ... + a_{2N}.x_N = b_2 \\ & ... \\ & a_{M1}.x_1 + a_{M2}.x_2 + ... + a_{MN}.x_N = b_M \end{align*}$
The scalars $a_{11}, ..., a_{MN}$ are the coefficients and the scalars $b_1, ..., b_M$ are the constants of the system of linear equations.
The solution for the set of linear equations is a sequence of scalars $n_1, n_2, ..., n_N$, such that, $x_1 = n_1, x_2 = n_2, ..., x_N = n_N$ that satisfies the system of linear equations.
Let us look at an example on how to solve a set of linear equations using Algebraic arithmetic.
Example-4 | Solve for the following three linear equations: $\begin{align*} & 2x + y - 2z = -3 \\ & x - 3y + z = 8 \\ & 4x - y - 2z = 3 \end{align*}$ |
---|---|
Let us first label the three linear equations so we can refer to them in the algebraic arithmetic operations. $\begin{flalign} & 2x + y - 2z = -3 \longrightarrow (1) \\ & x - 3y + z = 8 \longrightarrow (2) \\ & 4x - y - 2z = 3 \longrightarrow (3) \end{flalign}$ Performing the algebraic subtraction $(3) - (1)$, we get the following equation: $\begin{flalign} & 2x - 2y = 6 \longrightarrow (3) \end{flalign}$ Simplifying equation $(3)$, we get: $\begin{flalign} & x - y = 3 \longrightarrow (3) \end{flalign}$ Multiplying equation $(2)$ with scalar 2, we get: $\begin{flalign} & 2x - 6y + 2z = 16 \longrightarrow (2) \end{flalign}$ Performing the algebraic addition $(2) + (1)$, we get the following equation: $\begin{flalign} & 4x - 5y = 13 \longrightarrow (2) \end{flalign}$ Multiplying equation $(3)$ with scalar 4, we get: $\begin{flalign} & 4x - 4y = 12 \longrightarrow (3) \end{flalign}$ Performing the algebraic subtraction $(2) - (3)$, we can solve for $y$: $\begin{flalign} & y = -1 \end{flalign}$ Substituting for $y$ in equation $(3)$, we can solve for $x$: $\begin{flalign} & x = 2 \end{flalign}$ Substituting for $x$ and $y$ in equation $(1)$, we can solve for $z$: $\begin{flalign} & z = 3 \end{flalign}$ |
Things get complicated when the number of variables increases. This is where matrices come to the rescue. One could use matrices to represent the system of linear equations and find a solution using matrix operations. The process of finding the solution using the matrix operations is called Gaussain Elimination.
Using the system of three equations from the Example-4 above, the following is the matrix representation of the set of equations:
$\begin{bmatrix} 2 & 1 & -2 \\ 1 & -3 & 1 \\ 4 & -1 & -2 \end{bmatrix} \begin{bmatrix} x \\ y \\ z \end{bmatrix} = \begin{bmatrix} -3 \\ 8 \\ 3 \end{bmatrix}$
The above matrix equation is of the form: $\textbf{A}\vec{x} = \vec{b}$, where the matrix $\textbf{A}$ represents all the coefficients of the system of linear equations (referred to as the coefficient matrix), the column vector $\vec{x}$ represents all the variables in the system of linear equations, and the column vector $\vec{b}$ represents all the constants in the system of linear equations.
There will be situations when one (or more) of the variables may be missing from one or more equations - in those cases it implies that the corresponding coefficients are 0 (Zero).
Consider the following system of equations:
$\begin{align*} & x - 2y + 3z = 6 \\ & -x + y -z = -4 \\ & 2x - 3z = 3 \end{align*}$
Notice the $y$ term missing in the third equation. This simply means the coefficient of the $y$ term is 0.
The following is the matrix equivalent of the above set of linear equations:
$\begin{bmatrix} 1 & -2 & 3 \\ -1 & 1 & -1 \\ 2 & 0 & -3 \end{bmatrix} \begin{bmatrix} x \\ y \\ z \end{bmatrix} = \begin{bmatrix} 6 \\ -4 \\ 3 \end{bmatrix}$
In order to solve a set of linear equations using Gaussian elimination, a more compact form is used in practice, called the Augmented Matrix.
The following is the augmented matrix equivalent of the above set of linear equations:
$\begin{bmatrix} 1 & -2 & 3 &\bigm| & 6 \\ -1 & 1 & -1 &\bigm| & -4 \\ 2 & 0 & -3 &\bigm| & 3 \end{bmatrix}$
The process involved in the Gaussian elimination is to reduce the given augmented matrix to the following upper triangular matrix form (with 1s along the diagonal) through a series of row-wise matrix operations:
$\begin{bmatrix} 1 & a & b &\bigm| & p \\ 0 & 1 & c &\bigm| & q \\ 0 & 0 & 1 &\bigm| & r \end{bmatrix}$
In other words, the elements of the diagonal must be all ONEs and below the diagonal the elements must be all ZEROs.
This form of the upper triangular augmented matrix is called the Row Echelon form.
The following are the some of the properties of a matrix in row echelon form:
Rows containing all zeros elements MUST be at the bottom row of the matrix, below rows with non-zero element(s)
For rows that do not contain all zeros, the first non-zero element MUST be a 1
For rows that do not contain all zeros, the first non-zero number in each row is to the RIGHT of the first non-zero number in the row above it
In order to reduce a given matrix to a row echelon form, the following are the valid row-level matrix operations that one can perform on the given augmented matrix:
Interchange two rows. The notation used to indicate this operation is $R_i \leftrightarrow R_j$, where $i$ and $j$ are the corresponding row numbers
Multiplying a row with a non-zero scalar. The notation used to indicate this operation is $\beta R_i \rightarrow R_i$, where $i$ is the corresponding row number
Adding a multiple (can be negative) of a row to another row. The notation used to indicate this operation is $R_i + \beta R_j \rightarrow R_i$, where $i$ and $j$ are the corresponding row numbers
If we look at the operations we performed on the set of equations in the Example-4 above, it was exactly the same as the row-level matrix operations indicated above.
Let us look at an example on how to solve a set of linear equations using Gaussain Elimination.
Example-5 | Solve for the following three linear equations: $\begin{align*} & x - 2y + 3z = 9 \\ & -x + 3y = -4 \\ & 2x - 5y + 5z = 17 \end{align*}$ |
---|---|
Let us convert the three linear equations to an augmented matrix as follows: $\begin{bmatrix} 1 & -2 & 3 &\bigm| & 9 \\ -1 & 3 & 0 &\bigm| & -4 \\ 2 & -5 & 5 &\bigm| & 17 \end{bmatrix}$ Performing the operation $R_2 + R_1 \rightarrow R_2$, we get the following: $\begin{bmatrix} 1 & -2 & 3 &\bigm| & 9 \\ 0 & 1 & 3 &\bigm| & 5 \\ 2 & -5 & 5 &\bigm| & 17 \end{bmatrix}$ Performing the operation $R_3 + (-2)R_1 \rightarrow R_3$, we get the following: $\begin{bmatrix} 1 & -2 & 3 &\bigm| & 9 \\ 0 & 1 & 3 &\bigm| & 5 \\ 0 & -1 & -1 &\bigm| & -1 \end{bmatrix}$ Performing the operation $R_3 + R_2 \rightarrow R_3$, we get the following: $\begin{bmatrix} 1 & -2 & 3 &\bigm| & 9 \\ 0 & 1 & 3 &\bigm| & 5 \\ 0 & 0 & 2 &\bigm| & 4 \end{bmatrix}$ Performing the operation $\frac{1}{2}R_3 \rightarrow R_3$, we get the following: $\begin{bmatrix} 1 & -2 & 3 &\bigm| & 9 \\ 0 & 1 & 3 &\bigm| & 5 \\ 0 & 0 & 1 &\bigm| & 2 \end{bmatrix}$ From the augmented matrix just above, we can derive the solution for $z$, which equals $2$. From $R_2$, we know $y + 3z = 5$. Substituting $z = 2$, we can derive the solution for $y$, which equals $-1$. Finally, from $R_1$, we know $x - 2y + 3z = 9$. Substituting $y = -1$ and $z = 2$, we can derive the solution for $x$, which equals $1$. |
References