Honors 1: Undergraduate Math Lab^{∗}

Peter Sarnak^{†}, Steven Miller^{‡}, Alex Barnett^{§}

September 25th, 2002

Courant Institute of Mathematical Sciences New York University New York, NY

Abstract

We review some basic facts about Linear Algebra and Matrix Groups, and give an introduction to Random Matrix Theory. Lecture by Steven Miller; notes by Steven Miller and Alex Barnett.

1 Linear Algebra Review

Matrices can either be thought of as rectangular (often square) arrays of numbers, or as linear transformations from one space to another (or possible to the same space). The former picture is the simplest starting point, but as Professor Sarnak emphasized, it is the latter, geometric view that gives a deeper understanding.

To connect with the simpler vector case, a vector can be thought of as a list of real numbers which change in a certain way when the coordinate system changes, or as a geometric object with length and direction. The latter object is coordinate-independent, and has different representations in different choices of coordinate axes. Try to keep the geometric picture in mind for matrices.

∗Homepage: http://www.math.nyu.edu/∼millerj/

†E-mail: sarnak@math.princeton.edu

‡E-mail: millerj@cims.nyu.edu or sjmiller@math.princeton.edu

§E-mail: barnett@nmr.mgh.harvard.edu

1.1 Definitions

Given an n × m matrix A (where n is the number of rows and m is the number of columns), the transpose of A, denoted A^{T}, is the m × n matrix where the rows of A^{T} are the columns of A (or, equivalently, the columns of A^{T} are the rows of A).

Lemma 1.1 (AB)^{T} = B^{T} A^{T} and (A^{T})^{T} = A.

We leave the proof to the reader.

If an n×n matrix (also called a square matrix) A satisfies A^{T} = A, then we say A is symmetric

Example 1.2 Let A be the matrix

( \begin{matrix}2 & 2 & 4 & 2 \\ 1 & 1 & −2 & 2 \\ −2 & 0 & 0 & 1 \\ 1 & 1 & 2 & 1 \end{matrix} )  (1)

Then A^{T} is

( \begin{matrix}2 & 1 & −2 & 1 \\ 2 & 1 & 0 & 1 \\ 4 & −2 & 0 & 2 \\ 2 & 2 & 1 & 1 \end{matrix} )  (2)

Note the above matrix is not symmetric.

The number of degrees of freedom in a symmetric matrix (i.e. independent real numbers needed to completely specify the matrix) is n(n + 1)/2. Why? There are n^{2} entries, n on the diagonal. If you specify all entries above the diagonal and all entries on the diagonal, then you know the symmetric matrix.

There are n^{2}−n non-diagonal entries (half above the diagonal, half below). Thus, one needs to specify \frac{n^{2}−n}{2} + n = \frac{n^{2}+n}{2} entries.

Exercise 1.3 If A and B are symmetric, show AB is symmetric.

Matrix multiplication. We call the element in the i^{th} row and j^{th} column a_{ij}. Think of i = 1 ⋯ n going down the left side, and j = 1 ⋯ M going across the top. A vector v we represent as a column of elements with the i^{th} being v_{i}. A nice way to see matrix-vector multiplication is that the v_{i} give the coefficients by which the columns of A are linearly mixed together. For the product w = Av to make sense, the length (dimension) of v must equal m, and the dimension of w will be n. A is therefore a linear transform from m-dim space to n-dim space.

Multiple transformations appear written backwards: if we apply A then B then C to a vector, we write

w = CBAv.  (3)

Note that taking the product of two n × n matrices requires O(n^{3}) effort.

Exercise 1.4 Show that there are two ways to evaluate triple matrix products of the type CBA. The slow way involves O(n^{4}) effort. How about the fast way? How do these results scale for the case of a product of k matrices?

Definition 1.5 (Invertible Martices) A is invertible if a matrix B can be found such that BA = AB = I. The inverse is then written B = A^{−1}. Invertibility requires A to be square.

Transformations of a matrix. Just as with vectors, we can find out how the components of a square matrix A change under transformation. Say we have a scalar quantity x = w^{T} Av. We transform our coordinate system linearly such that the vector v has components v′ = Mv, where M is some invertible matrix representing the transformation. Therefore also w′ = Mw. The only way that x can remain unchanged by the transformation (as any scalar must), for all choices of v and w, is if the transformed matrix is written A′ = M^{−T} AM^{−1}. Check this via

x′ = w^{′T} A′v′ = (Mw)^{T} (M^{−T} AM^{−1})(Mv) = w^{T} IAIv = w^{T} Av = x. (4)

This is called a similarity transformation, or a conjugation. Really we have one object, the transformation A, but it may have different representations by a matrix of numbers, depending on the choice of basis.

Definition 1.6 (Orthogonal Matrices) Q is an orthogonal n × n matrix if it has real entries and Q^{T} Q = QQ^{T} = I.

Q is invertible, with inverse Q^{T}. The geometric meaning of Q is a rotation: the vector w = Qv is just v rotated (about the origin).

The number of degrees of freedom in an orthogonal matrix is n(n − 1)/2.

Exercise 1.7 In 3 dimensions a general rotation involves 3 angles (for example, azimuth, elevation, and ‘roll’). How many angles are needed in 4 dimensions? In 3d you rotate about a line-like axis (the set of points which do not move under rotation); what object do you rotate about in 4d?

Exercise 1.8 Show that the identity matrix I, always has representation I_{ij} = δ_{ij} regardless of the choice of basis. Hint: perform orthogonal tranformation on the matrix δ_{ij}.

The set of orthogonal matrices of order n forms a continuous (or topological) group, which we call O(n). (Not to be confused with “of order N”). Group properties:

• Associativity follows from that of matrix multiplication.

• The identity matrix acts as an identity element, since it is in the group.

• Inverse is the transpose (see above): Q^{−1} = Q^{T}.

• Closure is satisfied because any product QR of orthogonal matrices is itself orthogonal.

Exercise 1.9 Prove the last assertion.

However, not all the elements of O(n) can ‘talk’ to each other, i.e. you cannot reach all the elements by continuous transformation from the identity I.

Example for n = 2: a general order-2 orthogonal matrix can be written

( \begin{matrix}cos θ & − sin θ \\ sin θ & cos θ \end{matrix} ) or ( \begin{matrix}cos θ & − sin θ \\ − sin θ & − cos θ \end{matrix} ) ,  (5)

where 0 ≤ θ < 2π is a real angle. The first has determinant +1 and defines the ‘special’ (i.e. unit determinant) group SO(2) which is a subgroup of O(2) with identity I. The second has determinant −1 and corresponds to rotations with a reflection; this subgroup is disjoint from SO(2), and has the weird (reflecting) identity can be written in some basis as

( \begin{matrix}1 & 0 \\ 0 & −1 \end{matrix} ) .  (6)

Note that SO(2), alternatively written as the family of planar rotations R(θ), is isomorphic to the unit length complex numbers under the multiplication operation:

R(θ) ⟷ e^{iθ}.  (7)

Therefore we have R(θ_{1})R(θ_{2}) = R(θ_{1} + θ_{2}). This commutativity relation does not hold in higher n > 2.

Orthogonal transformations. If an orthogonal matrix Q is used for conjugation of a general square matrix A, then the rule Eq. 4 for transformation becomes,

A′ = QAQ^{T} .  (8)

This tells you how to ‘rotate’ a (square) matrix.

Definition 1.10 (Complex Conjugate Transpose) Let A be an n × m matrix. Then the complex conjugate transpose of A, denoted A^{∗}, is obtained by the following: (1) take the complex conjugate of A; ie, replace every entry a_{jk} = x_{jk} + iy_{jk} with \overline{a_{jk}} = x_{jk} − iy_{jk}, and call this matrix A_{1}; (2) take the transpose of A_{1}.

Exercise 1.11 Prove that (AB)^{∗} = B^{∗}A^{∗}.

Definition 1.12 (Dot or Inner Product) The dot (or inner) product of two real vectors v and w is defined as v^{T} w; if the vectors are complex, we instead use v^{∗}w.

Exercise 1.13 Show that the dot product is invariant under orthogonal transformation. That is, show that given two vectors, transforming them using the same orthogonal matrix leaves their dot product unchanged.

Definition 1.14 (Length of a vector) The length of a real vector v is |v|^{2} = v^{T} v; for a complex vector, we have |v|^{2} = v^{∗}v.

Definition 1.15 (Orthogonality) Two real vectors are orthogonal (also called perpendicular) if v^{T} w = 0; for two complex vectors, the equivalent condition is v^{∗}w = 0.

Definition 1.16 (Eigenvalue, Eigenvector) Recall λ is an eigenvalue and v is an eigenvector if Av = λv and v is not the zero vector.

Exercise 1.17 If v is an eigenvector of A with eigenvalue λ, show w = av, a ∈ ℂ, is also an eigenvector of A with eigenvalue λ.

Exercise 1.18 Show that given an eigenvalue λ and an eigenvector v, you can always find an eigenvector w with the same eigenvalue, but |w| = 1.

To find the eigenvalues, we solve the equation det(λI − A) = 0. This gives a polynomial p(λ) = 0. We call p(λ) the characteristic polynomial.

The trace of a matrix A, denote Tr(A) is the sum of the diagonal entries of A:

Tr(A) = ∑_{i=1}^{n} a_{ii}.  (9)

Lemma 1.19 Tr(A) = ∑_{i=1}^{n} λ_{i}.

The proof relies on writing out the characteristic equation and comparing powers of λ with the factorized version. By the fact that the polynomial has roots λ_{i} we can write

det(λI − A) = p(λ) = ∏_{i=1}^{n}(λ − λ_{i}).  (10)

Note the coefficient of λ^{n} is 1, thus we have ∏_{i}(λ−λ_{i}) and not c ∏_{i}(λ−λ_{i}) for some constant c.

By expanding out the RHS, the coefficient of λ^{n−1} is − ∑_{i=1}^{n} λ_{i}, which we will show is −Tr(A). Expanding the LHS, we want to find the corresponding coefficient in

| \begin{matrix}λ − a_{11} & −a_{12} & ⋯ & −a_{1n} \\ −a_{21} & λ − a_{22} \\ ⋮ & & ... \\ −a_{n1} & & & λ − a_{nn} \end{matrix} | .

We have to remember the expansion of the determinant. Taking the topleft-most 2 × 2 block, we see its determinant is (λ − a_{11})(λ − a_{22}) − a_{12}a_{21} = λ^{2} − (a_{11} + a_{22})λ + (a_{11}a_{22} − a_{12}a_{21}). The determinant of the top-left-most 3×3 block is then formed by (λ−a_{33}) times the above 2×2 determinant, plus two other multiples of determinants which can give only a highest power of λ of λ^{1}. Thus we see that the coefficient in λ^{2} is −(a_{11} + a_{22} + a_{33}). Repeating this argument for 4 × 4 block up to n × n gives us the coefficient of λ^{n−1} in the full determinant is − ∑_{i=1}^{n} a_{ii}. Since the LHS and RHS must be equal ∀λ, the LHS and RHS coefficients in λ^{n−1} are equal. □

Corollary 1.20 Tr(A) is invariant under rotation of basis.

The proof follows immediately from the invariance of the eigenvalues under rotation of basis. We need the following:

Lemma 1.21 det(AB) = det(A) det(B). Further, by induction one can show det(AB ⋯ Z) = det(A) det(B) ⋯ det(Z). Further, det(I) = 1.

Proof of Corollary: Let A = Q^{T} BQ. We show A and B have the same trace by showing A and B have the same eigenvalues. To find the eigenvalues of A we must solve:

det(λI − A) = det(λI − Q^{T} BQ)
= det(λQ^{T} Q − Q^{T} BQ)
= det(Q^{T} λIQ − Q^{T} BQ)
= det (Q^{T} (λI − B)Q)
= det(Q^{T}) det(λI − B) det(Q)
= det(Q^{T}) det(Q) det(λI − B)
= det(Q^{T} Q) det(λI − B) = det(I) det(λI − B) = det(λI − B).

(11)

As the eigenvalues of A and B satisfy the same equation, they are equal. □

1.2 Spectral Theorem for Real Symmetric Matrices

The main theorem we will prove is

Theorem 1.22 (Spectral Theorem) Let A be a real symmetric n×n matrix. Then there exists an orthogonal n × n matrix Q and a diagonal matrix Λ such that Q^{T} AQ = Λ. Moreover, the n eigenvalues of A are the diagonal entries of Λ.

This result is remarkable: it tells you that any real, symmetric matrix is diagonal when rotated into an appropriate basis (recall the rotation effect of conjugation using Q). In other words, the operation of matrix A on a vector v can be broken down into three steps:

Av = QΛQ^{T} v = (undo the rotation)(stretch along coord axes)(rotation)v.

(12)

Recall the ordering of transformations is read like Hebrew, right to left. The rotation is just the rotation into the basis of eigenvectors.

Furthermore, the eigenvalues λ_{i} (= diag els of Λ) are a set of numbers invariant under rotations of A. In other words, if A′ = P AP^{T} is an orthogonally-conjugated (i.e. P is orthogonal) version of A, then A′ has the same {λ_{i}} as A. Of course the ordering of the λ_{i} has to be chosen the same.

For the Spectral Theorem we prove a sequence of needed lemmas:

Lemma 1.23 The eigenvalues of a real symmetric matrix are real.

Let A be a real symmetric matrix with eigenvalue λ and eigenvector v. Note that we do not yet know that v has only real coordinates!

Therefore, Av = λv. Take the dot (or inner) product of both sides with the vector v^{∗}, the complex conjugate transpose of v:

v^{∗}Av = λv^{∗}v.  (13)

But the left hand side is real. The two sides are clearly complex numbers (ie, 1-dimensional matrices). Taking the complex conjugate transpose of the LHS gives

(v^{∗}(Av))^{∗} = (Av)^{∗}(v^{∗})^{∗} = v^{∗}Av.  (14)

Therefore, the LHS is real, implying the RHS is real. But clearly v^{∗}v is real (similar calculation). Thus, λ is real. □

We will only prove the Spectral Theorem when all the eigenvalues are distinct. Henceforth, we shall always assume A is a real symmetric matrix.

Lemma 1.24 The eigenvectors of a real symmetric matrix are real.

The eigenvalues solve the equation (λI −A)v = 0. Let λ be an eigenvalue. Then det(λI − A) = 0. Therefore the matrix B = λI − A is not invertible. Therefore it send a vector to 0 (standard linear algebra calculation).

Lemma 1.25 If λ_{1} and λ_{2} are two distinct eigenvalues of a real symmetric matrix A, then their corresponding eigenvectors are perpendicular.

We study v_{1}^{T} Av_{2}. Now

v_{1}^{T} Av_{2} = v_{1}^{T} (Av_{2}) = v_{1}^{T} (λ_{2}v_{2}) = λ_{2}v_{1}^{T} v_{2}.  (15)

Also,

v_{1}^{T} Av_{2} = v_{1}^{T} A^{T} v_{2} = (v_{1}^{T} A^{T})v_{2} = (Av_{1})^{T} v_{2} = (λ_{1}v_{1})^{T} v_{2} = λ_{1}v_{1}^{T} v_{2}. (16)

Therefore

λ_{2}v_{1}^{T} v_{2} = λ_{1}v_{1}^{T} v_{2} or (λ_{1} − λ_{2})v_{1}^{T} v_{2} = 0.  (17)

As λ_{1} ≠ λ_{2}, v_{1}^{T} v_{2} = 0. Thus, the eigenvectors v_{1} and v_{2} are perpendicular. □

We can now prove the Spectral Theorem for real symmetric matrices if there are n distinct eigenvectors.

Let λ_{1} to λ_{n} be the n distinct eigenvectors, and let v_{1} to v_{n} be the corresponding eigenvectors chosen so that each v_{i} has length 1.

Consider the matrix Q, where the first column of Q is v_{1}, the second column of Q is v_{2}, all the way to the last column of Q which is v_{n}:

Q = ( \begin{matrix}↑ & ↑ & & ↑ \\ v_{1} & v_{2} & ⋯ & v_{n} \\ ↓ & ↓ & & ↓ \end{matrix} )  (18)

The transpose of Q is

Q^{T} = ( ← v_{1} → ; ⋮ ; ← v_{n} → )  (19)

Exercise 1.26 Show that Q is an orthogonal matrix. Use the fact that the v_{i} all have length one, and are orthogonal (perpendicular) to each other.

Consider Q^{T} AQ. This is a matrix, call it B. To find its entry in the i^{th} row and j^{th} column, we look at

e_{i}^{T} Be_{j}  (20)

where the e_{k} are column vectors which are 1 in the k^{th} position and 0 elsewhere:

e_{k} = ( 0 ⋮ 0 1 0 ⋮ 0 )  (21)

Thus, we need only show that e_{i}^{T} Be_{j} = 0 if i ≠ j and equals λ_{j} if i = j.

Exercise 1.27 Show Qe_{i} = v_{i} and Q^{T} v_{i} = e_{i}.

We calculate

e_{i}^{T} Be_{j} = e_{i}^{T} Q^{T} AQe_{j}
= (e_{i}^{T} Q^{T})A(Qe_{j})
= (Qe_{i})^{T} A(Qe_{j})
= v_{i}^{T} Av_{j}
= v_{i}^{T} (Av_{j})
= v_{i}^{T} λ_{j}v_{j} = λ_{j}v_{i}^{T} v_{j}.  (22)

As v_{i}^{T} v_{j} equals 0 if i ≠ j and 1 if i = j, this proves the claim.

Thus, the off-diagonal entries of Q^{T} AQ are zero, and the diagonal entries are the eigenvalues λ_{j}. This shows that Q^{T} AQ is a diagonal matrix whose entries are the n eigenvalues of A. □

Note that, in the case of n distinct eigenvalues, not only can we write down the diagonal matrix, we can easily write down what Q should be. Further, by reordering the columns of Q, we see we reorder the positioning of the eigenvalues on the diagonal.