Introduction to Linear Algebra

The purpose of Eigen Decomposition is to extract two sets of features from a square matrix - eigenvalues, which are scalars denoted as $\lambda$, AND eigenvectors, which are vectors denoted as $\vec{v}$. The eigenvalues and eigenvectors are paired, meaning that each eigenvalue has an associated eigenvector. Given a square matrix of dimension M x M, there are M eigenvalues and M eigenvectors corresponding to each of the M column vectors in the matrix.

The eigenvalue $\lambda$ simply scales the eigenvector $\vec{v}$ that points in the same direction as $\vec{v}$.

The eigenvector is a special vector $\vec{v}$, such that, when it is transformed by some matrix $\textbf{A}$, the product $\textbf{A}\vec{v}$ points in the same direction as $\vec{v}$.

In other words, given the matrix $\textbf{A}$, the eigenvalue $\lambda$, and the eigenvector $\vec{v}$, they satisfy a very interesting condition: $\textbf{A}\vec{v} = \lambda\vec{v}$.

To discover the eigenvalues of a matrix, we will rewrite and simplify the equation as follows:

$\textbf{A}\vec{v} - \lambda\vec{v} = \textbf{0}$

That is, $\textbf{A}\vec{v} - \lambda\textbf{I}\vec{v} = \textbf{0}$

Or, $(\textbf{A} - \lambda\textbf{I})\vec{v} = \textbf{0}$

This implies that $\textbf{A} - \lambda\textbf{I} = \textbf{0}$

The above equation can be satisfied when the matrix $\textbf{A}$ is a sigular matrix (has linear dependence) and the determinant $\lvert\textbf{A} - \lambda\textbf{I}\rvert = 0$.

For eigen decomposition of a matrix, the first step is to identify the eigenvalues $\lambda$ and then use each of the eigenvalues to find the corresponding eigenvector $\vec{v}$.

The eigenvalues $\lambda$ will be unique, however, there will be many eigenvectors $\vec{v}$ corresponding to a eigenvalue, that satisfy the condition $\textbf{A}\vec{v} = \lambda\vec{v}$. The general rule here is to pick the one with a unit norm (or magnitude). That is, the eigenvector corresponding to a eigenvalue $\lambda$ is $\Large{\frac{\vec{v}}{\lVert\vec{v}\rVert}}$.

Let us look at an example to understand eigen decomposition.

Example-1	Find the eigenvalues and eigenvectors for the matrix: $\textbf{A} = \begin{bmatrix} 2 & 2 \\ 5 & -1 \end{bmatrix}$
To find the eigenvalues, we need to solve for $\lvert\textbf{A} - \lambda\textbf{I}\rvert = 0$. That is, $\begin{vmatrix} \begin{bmatrix} 2 & 2 \\ 5 & -1 \end{bmatrix} - \lambda \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} \end{vmatrix} = 0$ Or, $\begin{vmatrix} \begin{bmatrix} 2 & 2 \\ 5 & -1 \end{bmatrix} - \begin{bmatrix} \lambda & 0 \\ 0 & \lambda \end{bmatrix} \end{vmatrix} = 0$ Or, $\begin{vmatrix} 2 - \lambda & 2 \\ 5 & -1 - \lambda \end{vmatrix} = 0$ We know the determinant of $\begin{vmatrix} a & b \\ c & d \end{vmatrix} = ad - bc$ That is, $(2 - \lambda)(-1 - \lambda) - 10 = 0$ Or, $-2 -2\lambda + \lambda + {\lambda}^2 - 10 = 0$ Or, ${\lambda}^2 - \lambda - 12 = 0$ The above quadratic equation can be written as $(\lambda - 4)(\lambda + 3) = 0$ Hence, the eigenvalues are $\lambda = -3$ and $\lambda = 4$ We also know the relationship between eigenvalues and eigenvectors: $\textbf{A}\vec{v} = \lambda\vec{v}$ For $\lambda = -3$, $\begin{bmatrix} 2 & 2 \\ 5 & -1 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = -3\begin{bmatrix} x \\ y \end{bmatrix}$ That is, $\begin{align} & 2x + 2y = -3x \\ & 5x - y = -3y \end{align}$ Or, $x = -\frac{2}{5}y$ As we can observe, we can pick any value for $y$ and there is a corresponding value for $x$. Choosing $y = 5$, we get $x = -2$, and therefore the eigenvector would be $\vec{v} = \begin{bmatrix} -2 \\ 5 \end{bmatrix}$ Converting the eigenvector to a unit vector, we get the eigenvector would be $\vec{v} = \begin{bmatrix} -0.3714 \\ 0.9285 \end{bmatrix}$ Therefore, for the eigenvalue $\lambda = -3$, the corresponding eigenvector is $\begin{bmatrix} -0.3714 \\ 0.9285 \end{bmatrix}$ Similarly, for $\lambda = 4$, $\begin{bmatrix} 2 & 2 \\ 5 & -1 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = 4\begin{bmatrix} x \\ y \end{bmatrix}$ That is, $\begin{align} & 2x + 2y = 4x \\ & 5x - y = 4y \end{align}$ Or, $x = y$ Choosing $y = 1$, we get $x = 1$, and therefore the eigenvector would be $\vec{v} = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$ Converting the eigenvector to a unit vector, we get the eigenvector would be $\vec{v} = \begin{bmatrix} 0.7071 \\ 0.7071 \end{bmatrix}$ Therefore, for the eigenvalue $\lambda = 4$, the corresponding eigenvector is $\begin{bmatrix} 0.7071 \\ 0.7071 \end{bmatrix}$

Example-1

Find the eigenvalues and eigenvectors for the matrix: $\textbf{A} = \begin{bmatrix} 2 & 2 \\ 5 & -1 \end{bmatrix}$

To find the eigenvalues, we need to solve for $\lvert\textbf{A} - \lambda\textbf{I}\rvert = 0$.

That is, $\begin{vmatrix} \begin{bmatrix} 2 & 2 \\ 5 & -1 \end{bmatrix} - \lambda \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} \end{vmatrix} = 0$

Or, $\begin{vmatrix} \begin{bmatrix} 2 & 2 \\ 5 & -1 \end{bmatrix} - \begin{bmatrix} \lambda & 0 \\ 0 & \lambda \end{bmatrix} \end{vmatrix} = 0$

Or, $\begin{vmatrix} 2 - \lambda & 2 \\ 5 & -1 - \lambda \end{vmatrix} = 0$

We know the determinant of $\begin{vmatrix} a & b \\ c & d \end{vmatrix} = ad - bc$

That is, $(2 - \lambda)(-1 - \lambda) - 10 = 0$

Or, $-2 -2\lambda + \lambda + {\lambda}^2 - 10 = 0$

Or, ${\lambda}^2 - \lambda - 12 = 0$

The above quadratic equation can be written as $(\lambda - 4)(\lambda + 3) = 0$

Hence, the eigenvalues are $\lambda = -3$ and $\lambda = 4$

We also know the relationship between eigenvalues and eigenvectors: $\textbf{A}\vec{v} = \lambda\vec{v}$

For $\lambda = -3$, $\begin{bmatrix} 2 & 2 \\ 5 & -1 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = -3\begin{bmatrix} x \\ y \end{bmatrix}$

That is, $\begin{align*} & 2x + 2y = -3x \\ & 5x - y = -3y \end{align*}$

Or, $x = -\frac{2}{5}y$

As we can observe, we can pick any value for $y$ and there is a corresponding value for $x$.

Choosing $y = 5$, we get $x = -2$, and therefore the eigenvector would be $\vec{v} = \begin{bmatrix} -2 \\ 5 \end{bmatrix}$

Converting the eigenvector to a unit vector, we get the eigenvector would be $\vec{v} = \begin{bmatrix} -0.3714 \\ 0.9285 \end{bmatrix}$

Therefore, for the eigenvalue $\lambda = -3$, the corresponding eigenvector is $\begin{bmatrix} -0.3714 \\ 0.9285 \end{bmatrix}$

Similarly, for $\lambda = 4$, $\begin{bmatrix} 2 & 2 \\ 5 & -1 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = 4\begin{bmatrix} x \\ y \end{bmatrix}$

That is, $\begin{align*} & 2x + 2y = 4x \\ & 5x - y = 4y \end{align*}$

Or, $x = y$

Choosing $y = 1$, we get $x = 1$, and therefore the eigenvector would be $\vec{v} = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$

Converting the eigenvector to a unit vector, we get the eigenvector would be $\vec{v} = \begin{bmatrix} 0.7071 \\ 0.7071 \end{bmatrix}$

Therefore, for the eigenvalue $\lambda = 4$, the corresponding eigenvector is $\begin{bmatrix} 0.7071 \\ 0.7071 \end{bmatrix}$

Let us look at another example in the 3-dimensional space.

Example-2	Find the eigenvalues for the matrix: $\textbf{A} = \begin{bmatrix} -2 & -4 & 2 \\ -2 & 1 & 2 \\ 4 & 2 & 5 \end{bmatrix}$
To find the eigenvalues, we need to solve for $\lvert\textbf{A} - \lambda\textbf{I}\rvert = 0$. That is, $\begin{vmatrix} \begin{bmatrix} -2 & -4 & 2 \\ -2 & 1 & 2 \\ 4 & 2 & 5 \end{bmatrix} - \lambda \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \end{vmatrix} = 0$ Or, $\begin{vmatrix} \begin{bmatrix} -2 & -4 & 2 \\ -2 & 1 & 2 \\ 4 & 2 & 5 \end{bmatrix} - \begin{bmatrix} \lambda & 0 & 0 \\ 0 & \lambda & 0 \\ 0 & 0 & \lambda \end{bmatrix} \end{vmatrix} = 0$ Or, $\begin{vmatrix} -2-\lambda & -4 & 2 \\ -2 & 1-\lambda & 2 \\ 4 & 2 & 5-\lambda \end{vmatrix} = 0$ We know the determinant of $\begin{vmatrix} a & b & c \\ d & e & f \\ g & h & i \end{vmatrix} = a \begin{vmatrix} e & f \\ h & i \end{vmatrix} - b \begin{vmatrix} d & f \\ g & i \end{vmatrix} + c \begin{vmatrix} d & e \\ g & h \end{vmatrix} = a (ei - fh) - b (di - fg) + c (dh - eg)$ That is, $(-2 - \lambda)[(1-\lambda)(5-\lambda)-4] - (-4)[(-2)(5-\lambda)-8] + 2[-4-4(1-\lambda)] = 0$ Or, $(-2 - \lambda)[(1-6\lambda+{\lambda}^2] + 4[2\lambda-18] + 2[4\lambda-8] = 0$ Or, ${\lambda}^3 - 4{\lambda}^2 - 27\lambda + 90 = 0$ The above cubic equation can be written as $(\lambda - 3)({\lambda}^2 - \lambda - 30) = 0$ That is, $(\lambda - 3)(\lambda + 5)(\lambda - 6) = 0$ Hence, the eigenvalues are $\lambda = 3$, $\lambda = -5$, and $\lambda = 6$ We also know the relationship between eigenvalues and eigenvectors: $\textbf{A}\vec{v} = \lambda\vec{v}$ For $\lambda = 3$, $\begin{bmatrix} -2 & -4 & 2 \\ -2 & 1 & 2 \\ 4 & 2 & 5 \end{bmatrix} \begin{bmatrix} x \\ y \\ z \end{bmatrix} = 3\begin{bmatrix} x \\ y \\ z \end{bmatrix}$ That is, $\begin{align} & -2x - 4y + 2z = 3x \\ & -2x + y + 2z = 3y \\ & 4x + 2y + 5z = 3z \end{align}$ Rearranging terms, $\begin{align} & 5x + 4y - 2z = 0 \\ & x + y - z = 0 \\ & 2x + y + z = 0 \end{align}$ Solving for equations, we get $x = -\frac{2}{3}y$ Choosing $y = -3$, we get $x = 2$, and by substitution, we get $z = -1$, and therefore the eigenvector would be $\vec{v} = \begin{bmatrix} 2 \\ -3 \\ -1 \end{bmatrix}$ Converting the eigenvector to a unit vector, we get the eigenvector would be $\vec{v} = \begin{bmatrix} 0.5355 \\ -0.8018 \\ -0.2673 \end{bmatrix}$ Therefore, for the eigenvalue $\lambda = 3$, the corresponding eigenvector is $\begin{bmatrix} 0.5355 \\ -0.8018 \\ -0.2673 \end{bmatrix}$ Similarly, for $\lambda = -5$, $\begin{bmatrix} -2 & -4 & 2 \\ -2 & 1 & 2 \\ 4 & 2 & 5 \end{bmatrix} \begin{bmatrix} x \\ y \\ z \end{bmatrix} = -5\begin{bmatrix} x \\ y \\ z \end{bmatrix}$ That is, $\begin{align} & -2x - 4y + 2z = -5x \\ & -2x + y + 2z = -5y \\ & 4x + 2y + 5z = -5z \end{align}$ Rearranging terms, $\begin{align} & 3x - 4y + 2z = 0 \\ & x - 3y - z = 0 \\ & 2x + y + 5z = 0 \end{align}$ Solving for equations, we get $y = -z$ Choosing $z = -1$, we get $y = 1$, and by substitution, we get $x = 2$, and therefore the eigenvector would be $\vec{v} = \begin{bmatrix} 2 \\ 1 \\ -1 \end{bmatrix}$ Converting the eigenvector to a unit vector, we get the eigenvector would be $\vec{v} = \begin{bmatrix} 0.8165 \\ 0.4082 \\ -0.4082 \end{bmatrix}$ Therefore, for the eigenvalue $\lambda = -5$, the corresponding eigenvector is $\begin{bmatrix} 0.8165 \\ 0.4082 \\ -0.4082 \end{bmatrix}$ Finally, for $\lambda = 6$, $\begin{bmatrix} -2 & -4 & 2 \\ -2 & 1 & 2 \\ 4 & 2 & 5 \end{bmatrix} \begin{bmatrix} x \\ y \\ z \end{bmatrix} = 6\begin{bmatrix} x \\ y \\ z \end{bmatrix}$ That is, $\begin{align} & -2x - 4y + 2z = 6x \\ & -2x + y + 2z = 6y \\ & 4x + 2y + 5z = 6z \end{align}$ Rearranging terms, $\begin{align} & 4x + 2y - z = 0 \\ & 2x + 5y - 2z = 0 \\ & 4x + 2y - z = 0 \end{align}$ Solving for equations, we get $x = \frac{1}{6}y$ Choosing $y = 6$, we get $x = 1$, and by substitution, we get $z = 16$, and therefore the eigenvector would be $\vec{v} = \begin{bmatrix} 1 \\ 6 \\ 16 \end{bmatrix}$ Converting the eigenvector to a unit vector, we get the eigenvector would be $\vec{v} = \begin{bmatrix} 0.0584 \\ 0.3505 \\ 0.9347 \end{bmatrix}$ Therefore, for the eigenvalue $\lambda = 6$, the corresponding eigenvector is $\begin{bmatrix} 0.0584 \\ 0.3505 \\ 0.9347 \end{bmatrix}$

Example-2

Find the eigenvalues for the matrix: $\textbf{A} = \begin{bmatrix} -2 & -4 & 2 \\ -2 & 1 & 2 \\ 4 & 2 & 5 \end{bmatrix}$

To find the eigenvalues, we need to solve for $\lvert\textbf{A} - \lambda\textbf{I}\rvert = 0$.

That is, $\begin{vmatrix} \begin{bmatrix} -2 & -4 & 2 \\ -2 & 1 & 2 \\ 4 & 2 & 5 \end{bmatrix} - \lambda \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \end{vmatrix} = 0$

Or, $\begin{vmatrix} \begin{bmatrix} -2 & -4 & 2 \\ -2 & 1 & 2 \\ 4 & 2 & 5 \end{bmatrix} - \begin{bmatrix} \lambda & 0 & 0 \\ 0 & \lambda & 0 \\ 0 & 0 & \lambda \end{bmatrix} \end{vmatrix} = 0$

Or, $\begin{vmatrix} -2-\lambda & -4 & 2 \\ -2 & 1-\lambda & 2 \\ 4 & 2 & 5-\lambda \end{vmatrix} = 0$

We know the determinant of $\begin{vmatrix} a & b & c \\ d & e & f \\ g & h & i \end{vmatrix} = a \begin{vmatrix} e & f \\ h & i \end{vmatrix} - b \begin{vmatrix} d & f \\ g & i \end{vmatrix} + c \begin{vmatrix} d & e \\ g & h \end{vmatrix} = a (ei - fh) - b (di - fg) + c (dh - eg)$

That is, $(-2 - \lambda)[(1-\lambda)(5-\lambda)-4] - (-4)[(-2)(5-\lambda)-8] + 2[-4-4(1-\lambda)] = 0$

Or, $(-2 - \lambda)[(1-6\lambda+{\lambda}^2] + 4[2\lambda-18] + 2[4\lambda-8] = 0$

Or, ${\lambda}^3 - 4{\lambda}^2 - 27\lambda + 90 = 0$

The above cubic equation can be written as $(\lambda - 3)({\lambda}^2 - \lambda - 30) = 0$

That is, $(\lambda - 3)(\lambda + 5)(\lambda - 6) = 0$

Hence, the eigenvalues are $\lambda = 3$, $\lambda = -5$, and $\lambda = 6$

We also know the relationship between eigenvalues and eigenvectors: $\textbf{A}\vec{v} = \lambda\vec{v}$

For $\lambda = 3$, $\begin{bmatrix} -2 & -4 & 2 \\ -2 & 1 & 2 \\ 4 & 2 & 5 \end{bmatrix} \begin{bmatrix} x \\ y \\ z \end{bmatrix} = 3\begin{bmatrix} x \\ y \\ z \end{bmatrix}$

That is, $\begin{align*} & -2x - 4y + 2z = 3x \\ & -2x + y + 2z = 3y \\ & 4x + 2y + 5z = 3z \end{align*}$

Rearranging terms, $\begin{align*} & 5x + 4y - 2z = 0 \\ & x + y - z = 0 \\ & 2x + y + z = 0 \end{align*}$

Solving for equations, we get $x = -\frac{2}{3}y$

Choosing $y = -3$, we get $x = 2$, and by substitution, we get $z = -1$, and therefore the eigenvector would be $\vec{v} = \begin{bmatrix} 2 \\ -3 \\ -1 \end{bmatrix}$

Converting the eigenvector to a unit vector, we get the eigenvector would be $\vec{v} = \begin{bmatrix} 0.5355 \\ -0.8018 \\ -0.2673 \end{bmatrix}$

Therefore, for the eigenvalue $\lambda = 3$, the corresponding eigenvector is $\begin{bmatrix} 0.5355 \\ -0.8018 \\ -0.2673 \end{bmatrix}$

Similarly, for $\lambda = -5$, $\begin{bmatrix} -2 & -4 & 2 \\ -2 & 1 & 2 \\ 4 & 2 & 5 \end{bmatrix} \begin{bmatrix} x \\ y \\ z \end{bmatrix} = -5\begin{bmatrix} x \\ y \\ z \end{bmatrix}$

That is, $\begin{align*} & -2x - 4y + 2z = -5x \\ & -2x + y + 2z = -5y \\ & 4x + 2y + 5z = -5z \end{align*}$

Rearranging terms, $\begin{align*} & 3x - 4y + 2z = 0 \\ & x - 3y - z = 0 \\ & 2x + y + 5z = 0 \end{align*}$

Solving for equations, we get $y = -z$

Choosing $z = -1$, we get $y = 1$, and by substitution, we get $x = 2$, and therefore the eigenvector would be $\vec{v} = \begin{bmatrix} 2 \\ 1 \\ -1 \end{bmatrix}$

Converting the eigenvector to a unit vector, we get the eigenvector would be $\vec{v} = \begin{bmatrix} 0.8165 \\ 0.4082 \\ -0.4082 \end{bmatrix}$

Therefore, for the eigenvalue $\lambda = -5$, the corresponding eigenvector is $\begin{bmatrix} 0.8165 \\ 0.4082 \\ -0.4082 \end{bmatrix}$

Finally, for $\lambda = 6$, $\begin{bmatrix} -2 & -4 & 2 \\ -2 & 1 & 2 \\ 4 & 2 & 5 \end{bmatrix} \begin{bmatrix} x \\ y \\ z \end{bmatrix} = 6\begin{bmatrix} x \\ y \\ z \end{bmatrix}$

That is, $\begin{align*} & -2x - 4y + 2z = 6x \\ & -2x + y + 2z = 6y \\ & 4x + 2y + 5z = 6z \end{align*}$

Rearranging terms, $\begin{align*} & 4x + 2y - z = 0 \\ & 2x + 5y - 2z = 0 \\ & 4x + 2y - z = 0 \end{align*}$

Solving for equations, we get $x = \frac{1}{6}y$

Choosing $y = 6$, we get $x = 1$, and by substitution, we get $z = 16$, and therefore the eigenvector would be $\vec{v} = \begin{bmatrix} 1 \\ 6 \\ 16 \end{bmatrix}$

Converting the eigenvector to a unit vector, we get the eigenvector would be $\vec{v} = \begin{bmatrix} 0.0584 \\ 0.3505 \\ 0.9347 \end{bmatrix}$

Therefore, for the eigenvalue $\lambda = 6$, the corresponding eigenvector is $\begin{bmatrix} 0.0584 \\ 0.3505 \\ 0.9347 \end{bmatrix}$

In general, for an M x M matrix $\textbf{A}$, there will be M eigenvalues ($\lambda_1,\lambda_2,...,\lambda_M$) and their corresponding M eigenvectors ($\vec{v}_1,\vec{v}_2,...,\vec{v}_M$).

In other words,

$\begin{align*} & A\vec{v}_1 = \lambda_1\vec{v}_1 \\ & A\vec{v}_2 = \lambda_1\vec{v}_2 \\ & ... \\ & A\vec{v}_M = \lambda_M\vec{v}_M \end{align*}$.

The M eigenvectors ($\vec{v}_1,\vec{v}_2,...,\vec{v}_M$) can be succintly represented as a matrix $\textbf{V}$, where the columns in $\textbf{V}$ are the M eigenvectors ($\vec{v}_1,\vec{v}_2,...,\vec{v}_M$) respectively.

For example, for a 3 x 3 matrix, if the 3 eigenvectors are as follows:

$\vec{v}_1 = \begin{bmatrix} v_{11} \\ v_{21} \\ v_{31} \end{bmatrix}$

$\vec{v}_2 = \begin{bmatrix} v_{12} \\ v_{22} \\ v_{32} \end{bmatrix}$

$\vec{v}_3 = \begin{bmatrix} v_{13} \\ v_{23} \\ v_{33} \end{bmatrix}$

Then,

$\textbf{V} = \begin{bmatrix} v_{11} & v_{12} & v_{13} \\ v_{21} & v_{22} & v_{23} \\ v_{31} & v_{32} & v_{33} \end{bmatrix}$

Similarly, the M eigenvalues ($\lambda_1,\lambda_2,...,\lambda_M$) can be represented as a diagonal matrix $\Lambda$, where the M diagonal elements are the M eigenvalues ($\lambda_1,\lambda_2,...,\lambda_M$).

For example, for a 3 x 3 matrix, if the 3 eigenvalues are $\lambda_1, \lambda_2, \lambda_3$, then

$\Lambda = \begin{bmatrix} \lambda_1 & 0 & 0 \\ 0 & \lambda_2 & 0 \\ 0 & 0 & \lambda_3 \end{bmatrix}$

With that, the relationship between the M eigenvalues and the corresponding M eigenvectors of the matrix $\textbf{A}$ can be represented as: $\textbf{A}\textbf{V} = \textbf{V}\Lambda$.

NOTICE that the diagonal matrix of eigenvalues has to be on the right of $\textbf{V}$ as we want to apply the correct eigenvalue $\lambda$ to the corresponding column in $\textbf{V}$.

More precisely, $\Lambda\textbf{V} = \begin{bmatrix} \lambda_1.v_{11} & \lambda_1.v_{12} & \lambda_1.v_{13} \\ \lambda_2.v_{21} & \lambda_2.v_{22} & \lambda_2.v_{23} \\ \lambda_3.v_{31} & \lambda_3.v_{32} & \lambda_3.v_{33}\ \end{bmatrix}$, which is NOT correct.

While, $\textbf{V}\Lambda = \begin{bmatrix} \lambda_1.v_{11} & \lambda_2.v_{12} & \lambda_3.v_{13} \\ \lambda_1.v_{21} & \lambda_2.v_{22} & \lambda_3.v_{23} \\ \lambda_1.v_{31} & \lambda_2.v_{32} & \lambda_3.v_{33}\ \end{bmatrix}$, which IS correct.

Given $\textbf{A}\textbf{V} = \textbf{V}\Lambda$, it can be simplified as follows:

$\textbf{A}\textbf{V}\textbf{V}^{-1} = \textbf{V}\Lambda\textbf{V}^{-1}$

That is, $\textbf{A}\textbf{I} = \textbf{V}\Lambda\textbf{V}^{-1}$

Or, $\textbf{A} = \textbf{V}\Lambda\textbf{V}^{-1}$

Given the above equation, eigen decomposition is often referred to as Diagonalization.

For any square matrix $\textbf{A}$, how does one find the matrix power to $n$, that is, $\textbf{A}^n$ ??? For example, if $\textbf{A} = \begin{bmatrix} 2 & 1 \\ 1 & 3 \end{bmatrix}$, then to find $\textbf{A}^2$, one could perform a matrix multiplication such as $\textbf{A}^2 = \textbf{A}.\textbf{A} = \begin{bmatrix} 2 & 1 \\ 1 & 3 \end{bmatrix}.\begin{bmatrix} 2 & 1 \\ 1 & 3 \end{bmatrix} = \begin{bmatrix} 5 & 5 \\ 5 & 10 \end{bmatrix}$. Now, what about $\textbf{A}^5$ ??? For this, one could do it the laborious way using the matrix multiplication, that is, $\textbf{A}^5 = \textbf{A}.\textbf{A}.\textbf{A} .\textbf{A}.\textbf{A}$. The computation gets ugly when the matrix $\textbf{A}$ is of higher dimensions.

This is where eigen decomposition comes to the rescue.

The power of a diagonal matrix is easy to compute.

For example, let the diagonal matrix $\textbf{B} = \begin{bmatrix} 2 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & 2 \end{bmatrix}$.

Then, $\textbf{B}^2 = \textbf{B}.\textbf{B} = \begin{bmatrix} 2 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & 2 \end{bmatrix}. \begin{bmatrix} 2 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & 2 \end{bmatrix} = \begin{bmatrix} 4 & 0 & 0 \\ 0 & 4 & 0 \\ 0 & 0 & 4 \end{bmatrix}$.

Similarly, $\textbf{B}^3 = \textbf{B}.\textbf{B}.\textbf{B} = \begin{bmatrix} 2 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & 2 \end{bmatrix}. \begin{bmatrix} 2 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & 2 \end{bmatrix}.\begin{bmatrix} 2 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & 2 \end{bmatrix} = \begin{bmatrix} 8 & 0 & 0 \\ 0 & 8 & 0 \\ 0 & 0 & 8 \end{bmatrix}$.

In general, $\textbf{B}^n = \begin{bmatrix} 2^n & 0 & 0 \\ 0 & 2^n & 0 \\ 0 & 0 & 2^n \end{bmatrix}$.

We know the equation for eigen decomposition is $\textbf{A} = \textbf{V}\Lambda\textbf{V}^{-1}$.

Then, $\textbf{A}^3 = (\textbf{V}\Lambda\textbf{V}^{-1}).(\textbf{V}\Lambda\textbf{V}^{-1}).(\textbf{V}\Lambda\textbf{V}^{-1})$

That is, $\textbf{A}^3 = \textbf{V}\Lambda(\textbf{V}^{-1}\textbf{V})\Lambda(\textbf{V}^{-1}\textbf{V})\Lambda\textbf{V}^{-1}$

Or, $\textbf{A}^3 = \textbf{V}\Lambda\textbf{I}\Lambda\textbf{I}\Lambda\textbf{V}^{-1}$

Simplifying, we get $\textbf{A}^3 = \textbf{V}\Lambda\Lambda\Lambda\textbf{V}^{-1} = \textbf{V}\Lambda{^3}\textbf{V}^{-1}$

Therefore, in general $\textbf{A}^n = \textbf{V}\Lambda{^n}\textbf{V}^{-1}$.