Eigenvalues and Eigenvectors Few concepts to remember from linear - PowerPoint PPT Presentation

Eigenvalues and Eigenvectors

Few concepts to remember from linear algebra Let 𝑩 be an 𝑜×𝑛 matrix and the linear transformation 𝒛 = 𝑩𝒚 𝑩 𝒛 ∈ ℛ 𝒐 𝒚 ∈ ℛ 𝒏 → Rank: maximum number of linearly independent columns or rows of 𝑩 • Range 𝑩 = 𝒛 = 𝑩𝒚 ∀𝒚} • Null 𝑩 = 𝒚 𝑩𝒚 = 𝟏} •

Eigenvalue problem Let 𝑩 be an 𝑜×𝑜 matrix: 𝒚 ≠ 𝟏 is an eigenvector of 𝑩 if there exists a scalar 𝜇 such that 𝑩 𝒚 = 𝜇 𝒚 where 𝜇 is called an eigenvalue . If 𝒚 is an eigenvector, then α𝒚 is also an eigenvector. Therefore, we will usually seek for normalized eigenvectors , so that 𝒚 = 1 Note: When using Python, numpy.linalg.eig will normalize using p=2 norm.

How do we find eigenvalues? Linear algebra approach: 𝑩 𝒚 = 𝜇 𝒚 𝑩 − 𝜇 𝑱 𝒚 = 𝟏 Therefore the matrix 𝑩 − 𝜇 𝑱 is singular ⟹ 𝑒𝑓𝑢 𝑩 − 𝜇 𝑱 = 0 𝑞 𝜇 = 𝑒𝑓𝑢 𝑩 − 𝜇 𝑱 is the characteristic polynomial of degree 𝑜 . In most cases, there is no analytical formula for the eigenvalues of a matrix (Abel proved in 1824 that there can be no formula for the roots of a polynomial of degree 5 or higher) ⟹ Approximate the eigenvalues numerically !

Example 𝑩 = 2 1 𝑒𝑓𝑢 2 − 𝜇 1 2 − 𝜇 = 0 4 2 4 Solution of characteristic polynomial gives: 𝜇 . = 4, 𝜇 / = 0 To get the eigenvectors, we solve: 𝑩 𝒚 = 𝜇 𝒚 𝑦 $ 𝒚 = 1 2 − (4) 1 = 0 𝑦 % 2 4 2 − (4) 0 𝑦 $ 2 − (0) 1 = 0 𝒚 = −1 𝑦 % 4 2 − (0) 0 2 Notes: The matrix 𝑩 is singular (det(A)=0), and rank( 𝑩 )=1 The matrix has two distinct real eigenvalues The eigenvectors are linearly independent

Diagonalizable Matrices A 𝑜×𝑜 matrix 𝑩 with 𝑜 linearly independent eigenvectors 𝒗 is said to be diagonalizable . 𝑩 𝒗 𝟐 = 𝜇 . 𝒗 𝟐 , 𝑩 𝒗 𝟑 = 𝜇 / 𝒗 𝟑 , … 𝑩 𝒗 𝒐 = 𝜇 ; 𝒗 𝒐 , In matrix form: 𝜇 # 0 0 𝑩 𝒗 𝟐 … 𝒗 𝒐 = 𝜇 # 𝒗 𝟐 𝜇 $ 𝒗 𝒐 = 𝒗 𝟐 … 𝒗 𝒐 … 0 ⋱ 0 0 0 𝜇 $ This corresponds to a similarity transformation 𝑩𝑽 = 𝑽𝑬 ⟺ 𝑩 = 𝑽𝑬𝑽 %𝟐

𝑒𝑓𝑢 2 − 𝜇 1 𝑩 = 2 1 Example 2 − 𝜇 = 0 4 4 2 Solution of characteristic polynomial gives: 𝜇 . = 4, 𝜇 / = 0 To get the eigenvectors, we solve: 𝑩 𝒚 = 𝜇 𝒚 𝑦 $ 𝒚 = 0.447 2 − (4) 1 = 0 𝒚 = 1 or normalized 𝑦 % 0.894 4 2 − (4) 0 2 eigenvector ( 𝑞 = 2 norm) 𝑦 $ 𝒚 = −0.447 2 − (0) 1 𝒚 = −1 = 0 𝑦 % 0.894 2 4 2 − (0) 0 𝑽 = 0.447 −0.447 𝑬 = 4 0 𝑩 = 𝑽𝑬𝑽 <. 0.894 0.894 0 0 Notes: The matrix 𝑩 is singular (det(A)=0), and rank( 𝑩 )=1 Since 𝑩 has two linearly independent eigenvectors, the matrix 𝑽 is full rank, and hence, the matrix 𝑩 is diagonalizable.

Example The eigenvalues of the matrix: 𝑩 = 3 −18 2 −9 are 𝜇 . = 𝜇 / = −3 . Select the incorrect statement: A) Matrix 𝑩 is diagonalizable B) The matrix 𝑩 has only one eigenvalue with multiplicity 2 C) Matrix 𝑩 has only one linearly independent eigenvector D) Matrix 𝑩 is not singular

Let’s look back at diagonalization… 1) If a 𝑜×𝑜 matrix 𝑩 has 𝑜 linearly independent eigenvectors 𝒚 then 𝑩 is diagonalizable, i.e., 𝑩 = 𝑽𝑬𝑽 <𝟐 where the columns of 𝑽 are the linearly independent normalized eigenvectors 𝒚 of 𝑩 (which guarantees that 𝑽 <𝟐 exists) and 𝑬 is a diagonal matrix with the eigenvalues of 𝑩 . 2) If a 𝑜×𝑜 matrix 𝑩 has less then 𝑜 linearly independent eigenvectors, the matrix is called defective (and therefore not diagonalizable). 3) If a 𝑜×𝑜 symmetric matrix 𝑩 has 𝑜 distinct eigenvalues then 𝑩 is diagonalizable.

A 𝒐×𝒐 symmetric matrix 𝑩 with 𝒐 distinct eigenvalues is diagonalizable. Suppose 𝜇 , 𝒗 and 𝜈, 𝒘 are eigenpairs of 𝑩 𝜇 𝒗 = 𝑩𝒗 𝜈 𝒘 = 𝑩𝒘 𝜇 𝒗 = 𝑩𝒗 → 𝒘 1 𝜇 𝒗 = 𝒘 1 𝑩𝒗 𝜇 𝒘 1 𝒗 = 𝑩 𝑼 𝒘 1 𝒗 = 𝑩 𝒘 1 𝒗 = 𝜈 𝒘 1 𝒗 → 𝜈 − 𝜇 𝒘 1 𝒗 = 0 If all 𝑜 eigenvalues are distinct → 𝜈 − 𝜇 ≠ 0 Hence, 𝒘 1 𝒗 = 0 , i.e., the eigenvectors are orthogonal (linearly independent), and consequently the matrix 𝑩 is diagonalizable. Note that a diagonalizable matrix 𝑩 does not guarantee 𝑜 distinct eigenvalues.

Some things to remember about eigenvalues: • Eigenvalues can have zero value • Eigenvalues can be negative • Eigenvalues can be real or complex numbers • A 𝑜×𝑜 real matrix can have complex eigenvalues • The eigenvalues of a 𝑜×𝑜 matrix are not necessarily unique. In fact, we can define the multiplicity of an eigenvalue. • If a 𝑜×𝑜 matrix has 𝑜 linearly independent eigenvectors, then the matrix is diagonalizable

How can we get eigenvalues numerically? Assume that 𝑩 is diagonalizable (i.e., it has 𝑜 linearly independent eigenvectors 𝒗 ). We can propose a vector 𝒚 which is a linear combination of these eigenvectors: 𝒚 = 𝛽 # 𝒗 # + 𝛽 ' 𝒗 ' + ⋯ + 𝛽 $ 𝒗 $ Then we evaluate 𝑩 𝒚 : 𝑩 𝒚 = 𝛽 # 𝑩𝒗 # + 𝛽 ' 𝑩𝒗 ' + ⋯ + 𝛽 $ 𝑩𝒗 $ And since 𝑩𝒗 # = 𝜇 # 𝒗 # we can also write: 𝑩 𝒚 = 𝛽 # 𝜇 # 𝒗 # + 𝛽 ' 𝜇 ' 𝒗 ' + ⋯ + 𝛽 $ 𝜇 $ 𝒗 $ where 𝜇 ( is the eigenvalue corresponding to eigenvector 𝒗 ( and we assume |𝜇 # | > |𝜇 ' | ≥ |𝜇 ) | ≥ ⋯ ≥ |𝜇 $ |

Power Iteration Our goal is to find an eigenvector 𝒗 ( of 𝑩. We will use an iterative process, where we start with an initial vector, where here we assume that it can be written as a linear combination of the eigenvectors of 𝑩 . 𝒚 * = 𝛽 # 𝒗 # + 𝛽 ' 𝒗 ' + ⋯ + 𝛽 $ 𝒗 $ And multiply by 𝑩 to get: 𝒚 # = 𝑩 𝒚 * = 𝛽 # 𝜇 # 𝒗 # + 𝛽 ' 𝜇 ' 𝒗 ' + ⋯ + 𝛽 $ 𝜇 $ 𝒗 $ 𝒚 ' = 𝑩 𝒚 # = 𝛽 # 𝜇 # ' 𝒗 # + 𝛽 ' 𝜇 ' ' 𝒗 ' + ⋯ + 𝛽 $ 𝜇 $ ' 𝒗 $ ⋮ 𝒚 + = 𝑩 𝒚 +%# = 𝛽 # 𝜇 # + 𝒗 # + 𝛽 ' 𝜇 ' + 𝒗 ' + ⋯ + 𝛽 $ 𝜇 $ + 𝒗 $ Or rearranging… + + 𝜇 ' 𝜇 $ 𝒚 + = 𝜇 # + 𝛽 # 𝒗 # + 𝛽 ' 𝒗 ' + ⋯ + 𝛽 $ 𝒗 $ 𝜇 # 𝜇 #

Power Iteration B B 𝜇 / 𝜇 ; 𝒚 B = 𝜇 . B 𝛽 . 𝒗 . + 𝛽 / 𝒗 / + ⋯ + 𝛽 ; 𝒗 ; 𝜇 . 𝜇 . Assume that 𝛽 . ≠ 0 , the term 𝛽 . 𝒗 . dominates the others when 𝑙 is very large. B Since |𝜇 . > |𝜇 / , we have C ! ≪ 1 when 𝑙 is large C " Hence, as 𝑙 increases, 𝒚 B converges to a multiple of the first eigenvector 𝒗 . , i.e., C " # = 𝛽 . 𝒗 . or 𝒚 B → 𝛽 . 𝜇 . B 𝒗 . 𝒚 # lim B→D

How can we now get the eigenvalues? If 𝒚 is an eigenvector of 𝑩 such that 𝑩 𝒚 = 𝜇 𝒚 then how can we evaluate the corresponding eigenvalue 𝜇 ? 𝜇 = 𝒚 𝑼 𝑩𝒚 Rayleigh coefficient 𝒚 𝑼 𝒚

Normalized Power Iteration & & 𝜇 % 𝜇 ' 𝒚 & = 𝜇 $ & 𝛽 $ 𝒗 $ + 𝛽 % 𝒗 % + ⋯ + 𝛽 ' 𝒗 ' 𝜇 $ 𝜇 $ 𝒚 𝟏 = arbitrary nonzero vector 𝒚 𝟏 𝒚 𝟏 = 𝒚 𝟏 for 𝑙 = 1,2, … 𝒛 B = 𝑩 𝒚 B<. 𝒛 # 𝒚 B = 𝒛 #

Normalized Power Iteration B B 𝜇 / 𝜇 ; 𝒚 B = 𝜇 . B 𝛽 . 𝒗 . + 𝛽 / 𝒗 / + ⋯ + 𝛽 ; 𝒗 ; 𝜇 . 𝜇 . What if the starting vector 𝒚 𝟏 have no component in the dominant eigenvector 𝒗 $ ( 𝛽 $ = 0 )? Demo “Power Iteration

Normalized Power Iteration B B 𝜇 / 𝜇 ; 𝒚 B = 𝜇 . B 𝛽 . 𝒗 . + 𝛽 / 𝒗 / + ⋯ + 𝛽 ; 𝒗 ; 𝜇 . 𝜇 . What if the first two largest eigenvalues (in magnitude) are the same, |𝜇 $ = |𝜇 % ? B B 𝒚 B = 𝜇 . B 𝛽 . 𝒗 . + 𝜇 . B 𝜇 / 𝜇 ; 𝛽 / 𝒗 / + 𝜇 . B … + 𝛽 ; 𝒗 ; 𝜇 . 𝜇 . Demo “Power Iteration

Potential pitfalls Starting vector 𝒚 𝟏 may have no component in the dominant eigenvector 𝒗 " (𝛽 " = 1. 0) . This is usually unlikely to happen if 𝒚 𝟏 is chosen randomly, and in practice not a problem because rounding will usually introduce such component. 2. Risk of eventual overflow (or underflow): in practice the approximated eigenvector is normalized at each iteration (Normalized Power Iteration) First two largest eigenvalues (in magnitude) may be the same: |𝜇 " | = |𝜇 # | . In this 3. case, power iteration will give a vector that is a linear combination of the corresponding eigenvectors: • If signs are the same, the method will converge to correct magnitude of the eigenvalue. If the signs are different, the method will not converge. • This is a “real” problem that cannot be discounted in practice.

Error B B 𝜇 / 𝜇 ; 𝒚 B = 𝜇 . B 𝛽 . 𝒗 . + 𝛽 / 𝒗 / + ⋯ + 𝛽 ; 𝒗 ; 𝜇 . 𝜇 . 𝐹𝑠𝑠𝑝𝑠 We can see from the above that the rate of convergence depends on the ratio C ! C " , that is: B 𝜇 / 𝜇 . <B 𝒚 B − 𝛽 . 𝒗 . = 𝑃 𝜇 .

Convergence and error B 𝒚 B = 𝒗 . + 𝛽 / 𝜇 / 𝒗 / + ⋯ 𝛽 . 𝜇 . 𝒇 & 𝒇 PQR 𝒇 P ≈ ? S ? R Power method has linear convergence, which is quite slow.

Iclicker question A) 0.1536 B) 0.192 C) 0.09 D) 0.027