For $N =4$, we can attempt to build a minimizer to the solve the system $Ax = b$. In order to find an $x$ such that $Ax = b$, we could define a minimizer $\phi(x)$ such that:
\[\phi(x)=\|b - Ax\|^2\]
Here $\phi(x)=0$ would mean that $x$ is an exact solution of the system $Ax = b$. We can then attempt to write such minimizer for $N =4$:
As it can be seen from the Hessian calculation, the Hessian is positive definite forall $x$s. This means, by the sufficient condition of minimizers, that we can find a minimizer by solving $\Delta\phi(x)=0$ (i.e. finding stationary points in the hyperspace defined by $\phi(x)$. Solving that, we find:
which is indeed the minimizer and solution of $Ax = b$. Therefore, $\phi(x)$ is a valid energy function. Although the $\phi(x)$ given here is not strictly in a matrix vector product quadratic form, it is indeed a valid energy function and for different values of $N$ similar fashioned $\phi(x)$ could be derived. Therefore, we can say that the problem has an energy function.
As by the definitions of $A$ and $b$ given in the assignment, we already enforce $x_1= x_n =0$, since the first and the last term of the matrix-vector product $Ax$ are $x_1$ and $x_n$ respectively and the system of equations represented by $Ax = b$ would indeed include the equalities $x_1= b_1=0$ and $x_n = b_n =0$. Therefore, we can simply alter the matrix $A$ without any need to perform any other transformation in the system.
We therefore define $\overline{A}$ as a copy of $A$ where $\overline{A}_{2,1}=\overline{A}_{n-1, n}=0$.
Due to space constraints, the $R^{1000}$ column vector for the solution of $x$ is not shown here but can be easily derived by running the script and checking the variable \texttt{x} after the script execution.
We can say that the Conjugate Gradient method is a Krylov subspace method since we can say that both all residuals $r_i$ and all search directions $p_i$ are contained in the span of $0$ to $k$ repeated applications of matrix transformer $A$ onto the initial search direction $p_0$. Broadly speaking, this property is directly connected with the fact that the CG methods performs up to $n$ steps exactly for a $n$ by $n$ matrix, walking steps that are all $A$-orthogonal with each other (i.e. for all couples of search steps $p_i$, $p_j$ where $i \neq j$, $\langle Ap_i, p_j \rangle=0$).
To sum up our claim, we can say CG is indeed a Krylov subspace method because:
\[\text{span}\{r_0, r_1, \ldots, r_n\}=\text{span}\{r_0, A r_0, \ldots, A^k r_0\}\]
\[\text{span}\{p_0, p_1, \ldots, p_n\}=\text{span}\{r_0, A r_0, \ldots, A^k r_0\}\]
These statements have been already proven in class and the proof can be found in Theorem 5.3 of Nocedal's book.
% These statements can be proven by induction over $k$. The base case holds trivially for $k=0$, while we have by the induction hypothesis for any $k$ that:
% \[r_k \in \text{span}\{r_0, A r_0, \ldots, A^k r_0\}\;\;\; p_k \in \text{span}\{r_0, A r_0, \ldots, A^k r_0\}\]
% We'd like to prove that the two properties hold from $k+1$ starting from the hypothesis on $k$. We first multiply the first hypothesis by $A$ from the left:
% We need to add $r_0$ in the span again since one of the components that defines $r_1$ is indeed $r_0$. We don't need to add other terms in the span since $r_1$ to $r_k$ are in the span already by the induction hypothesis.
% Combining this expression with the induction hypothesis we prove induction of the first statement by having:
Below are the chosen $b$ vector (which is unchanged between executions due to fixing the random generator seed in the script) and the solutions of $x$ for each matrix respectively.
The code to generate the plots below and to compute the logarithm energy norm of the error can be found in section 2.3 of the \texttt{main.m} MATLAB script.
