NOTES ON LINEAR ALGEBRA

 

CONTENTS:

[12] EIGENVALUES / EIGENVECTORS

 

 

[12] EIGENVALUES / EIGENVECTORS

 

In the last section, we examined when two vectors are a basis for the plane. Let’s recall what this means. The plane is 2-dimensional. So, we expect that we should be able to specify any vector with two pieces of information, say East component and North component. This corresponds to the standard basis (x-axis, y-axis).

 

In [11] we saw that, as long as W1 and W2 are not in the same direction, they are a basis for the plane. This means we can write any vector V as V = a W1 + b W2, where a and b are numbers that can be determined, and depend on V.

 

However, nowhere in [11] did we discuss why we would want to use a basis other than the standard x-axis, y-axis.

 

The reason is geometry. Often we’ll be studying certain matrices that model physical systems. Those physical systems may have certain axes of symmetry, which will often manifest itself in the matrix. And what we will find is that the matrix looks more ‘natural’, more ‘symmetric’, if we change basis.

 

Let A be a matrix acting on a vector v. What can we say about the vector Av?

 

In general, not much, unless I give you information about A and v. A vector encodes two pieces of information: magnitude, and direction. When we apply a matrix to a vector, we get a new vector. Usually, that vector will have a different magnitude and a different direction.

 

In terms of computations, this is often unfortunate. We may be interested in some iterative system, where we might have A100 v or A6022045 v. If Av is in a different direction than v, we have no quick and easy way to calculate A2 v. Why? We know the magnitude and direction of Av. But we know nothing about A(Av).

 

If Av is in the same direction as v, however, it’s a different ballgame. Let’s say Av = 3v. Then we can calculate Aany power v easily.

 

 

 

For example:

 

A2 v     =          A (Av)

            =          A (3 v)

            =          3 (A v)

            =          3 (3 v)

            =          32 v

 

 

A3 v     =          A (A2 v)

            =          A (32 v)            by the previous calculation

            =          32 (A v)

            =          32 (3 v)

            =          33 v

 

 

A4 v     =          A (A3 v)

            =          A (33 v)            by the previous calculation

            =          33 (A v)

            =          33 (3 v)

            =          34 v

 

A5 v     =          A (A4 v)

            =          A (34 v)            by the previous calculation

            =          34 (A v)

            =          34 (3 v)

            =          35 v

 

 

And a similar argument shows A6 = 36 v, ..., An v = 3n v.

 

 


DEFINITION OF EIGENVALUE / EIGENVECTOR:

We say v is an eigenvector and l is its eigenvalue if

(1) A v = l v

(2) v is not the zero vector

 

 

 

 

 

 

 

Note that a vector is an eigenvector relative to a given matrix. For example,

 

(2 1) (1)                       (3)                       (1)

(1 2) (1)           =          (3)        =          3 (1)

 

(2 1) (1)                       (1)                       (1)

(1 2) (-1)          =          (-1)      =          1 (-1)

 

So (1,1) is an eigenvector with eigenvalue 3, while (1,-1) is an eigenvector with eigenvalue 1. But if we consider a different matrix

 

(1 2) (1)                       (2)

(3 4) (1)           =          (7)

 

we see (1,1) is not an eigenvector.

 

Why do we exclude the zero vector? The reason is the zero vector would be an eigenvector for ANY matrix, and ANY number would be its eigenvalue. Let Z be the zero vector. Then

 

            A Z = Z = 2 Z = 3 Z = -2342.324 Z

 

and Z would not have a unique eigenvalue. Again, this is a matter of notation / convenience. We will see later its just easier if all eigenvalues are non-zero, for we’ll prove certain nice facts about them. For example, a beautiful theorem of Linear Algebra (The Principal Axis Theorem) states that if you have a symmetric real matrix, you can find a basis of mutually perpendicular vectors Vi such that each Vi is an eigenvector of the matrix A! Wow! This means we can compute the action of large powers of symmetric matrices with very little work.

 

Let’s see now how eigenvectors can make life liveable. We’ll work with the matrix

 

           

            A         =          (2 1)

                                    (1 2)

 

We saw above that if V = (1,1) and W = (1,-1) then

 

            A V = 3 V

            A W = 1 W

 

V and W are not in the same direction, so they’re a basis for the plane. So, if you give me any vector (x,y), I can find a and b (depending on x and y) such that

 

            (x,y) = a V + b W.

 

Why is this helpful?

 

Let’s build up in stages.

 

            A(V + W)        =          AV + AW

                                    =          3V + 1W         =          3V + W

 

            A(2V + 11 W)             =          A(2V)  + A(11 W)

                                                =          2 (A V) + 11 (A W)

                                                =          2 (3 V) + 11 W

                                                =          6 V + 11 W

 

            A4 (2V + 11 W)           =          B (2 V + 11 W),          where B = A4

                                                =          B (2V) + B (11 W)

                                                =          2 (B V) + 11 (B W)

                                                =          2 (A4 V) + 11 (A4 W)

                                                =          2 (34 V) + 11 W

 

NOTE: we have the rule

 

            A (X + Y)                    =          A X + AY

 

This is still true for powers of A, as A4, A5, etc. are also matrices:

 

            A4 (X + Y)                   =          A4 X + A4 Y

 

 

So, in full generality:

 

Eq 12.1            A n (aV + bW)             =          An (aV) + An (bW)

                                                            =          a (An V) + b (An W)

                                                            =          a (3n V) + b W

 

So we see how easy it is to calculate. Let’s take n = 2,000,000, and consider A U, where U = (2,3). Now, A U is not in the same direction as U, nor is A2 U in the same direction as U or A U, etc., so we have 2,000,000 matrix multiplications to do! That’s a lot.

 

The other way is, FIRST, we express U in terms of our nice basis V, W. We do this by Gaussian Elimination:

 

            U = a V + b W

 

            (2)                       (1)                   (1)

            (3)        =          a (1)     +          b(-1)

 

            (2)                    (1a)                  (1b)

            (3)        =          (1a)      +          (-1b)

 

 

            (2)                    (1a +  1b)

            (3)        =          (1a + -1b)

 

 

            (2)                    (1  1) (a)

            (3)        =          (1 -1) (b)

 

 

And then we do Gaussian Elimination. And now we are done! By Eq 12.1 we are done – all we have to do is put in the values of a and b, and that n = 2,000,000.

 

Doing the calculation this way is about two steps; the other way is 2,000,000. This is a phenomenal savings. The long way is beyond the strength of the computers -–the numbers are just too large.

 

Why are we able to have such a savings? This is not a trivial point – it’s one of the strengths of Linear Algebra. Linear Algebra is an efficient way of arranging computations. The long way involves lots of hidden cancellation, cancellation that never survives to the end when we group everything.

 

An example might help. Consider doing the following addition:

 

             1

            +5        - 5

                        +6        -6

                                    +7        -7

                                                +8        -8

                                                            +9        -9

 

If we add horizontally, each row is zero but the first. It doesn’t matter how many rows I’ve got, the final answer is just going to be one. There’s only one computation to do.

 

What if we add vertically?

 

The we get 6 + 1 +1 + 1 + 1 - 9 = 1. We get the same answer, but it’s many more steps. The reason is we add 5 then subtract 5. We add 6 then subtract 6. We add 7 then subtract 7. We add 8 then subtract 8. We add 9 then subtract 9. We keep doing things that cancel, but we don’t realize it, and hence have to go thru all the steps.