\serieheader{Numerical Computing}{2020}{Student: Claudio Maggioni}{Discussed with: --}{Solution for Project 1}{}
\newline
\assignmentpolicy
The purpose of this assignment\footnote{This document is originally based on a SIAM book chapter from \textsl{Numerical Computing with Matlab} from Clever B. Moler.} is to learn the importance of numerical linear algebra algorithms to solve fundamental linear algebra problems that occur in search engines.
\subsubsection{What assumptions should be made to guarantee convergence of the power method?}
The first assumption to make is that the biggest eigenvalue in terms of absolute values should (let's name it $\lambda_1$)
be strictly greater than all other eigenvectors, so:
$$|\lambda_1| < |\Lambda_i| \forall i \in\{2..n\}$$
Also, the eigenvector \textit{guess} from which the power iteration starts must have a component in the direction of $x_i$, the eigenvector for the eigenvalue $\lambda_1$ from before.
\subsubsection{What is a shift and invert approach?}
The shift and invert approach is a variant of the power method that may significantly increase the rate of convergence where some application of the vanilla method require large numbers of iterations. This improvement is achieved by taking the input matrix $A$ and deriving a matrix $B$ defined as:
\[B =(A -\alpha I)^{-1}\]
where $\alpha$ is an arbitrary constant that must be chosen wisely in order to increase the rate of convergence. Since the eigenvalues $u_i$ of B can be derived from the eigenvalues $\lambda_i$ of A, namely:
$$u_i =\frac{1}{\lambda_i -\alpha}$$
the rate of convergence of the power method on B is:
By choosing $\alpha$ close to $\lambda_1$, the convergence is sped up. To further increase the rate of convergence (up to a cubic rate), a new $\alpha$, and thus a new $B$, may be chosen for every iteration.
\subsubsection{What is the difference in cost of a single iteration of the power method, compared to the inverse iteration?}
Inverse iteration is generally more expensive than a regular application of the power method, due to the overhead caused by the intermediate matrix B. One must either recompute B every time $\alpha$ changes, which is rather expensive due to the inverse operation in the definition of B, or one must solve the matrix equation $(A -\alpha I)v_k = v_{k-1}$ in every iteration.
\subsubsection{What is a Rayleigh quotient and how can it be used for eigenvalue computations?}
The Railegh quotient is an effective way to either compute the corresponding eigenvalue of an eigenvector or the corresponding eigenvalue approximation of an eigenvector approximation. I.e., if $x$ is an eigenvector, then:
\[\lambda=\mu(x)=\frac{x^T A x}{x^T x}\]
is the corresponding eigenvalue, while if $x$ is an eigenvector approximation, for example found through some iterations of the power method, then $\lambda$ is the closest possible approximation to the corresponding eigenvalue in a least-square sense.
