midterm: done 75% of 3

Claudio_Maggioni_midterm/Claudio_Maggioni_midterm.md

@@ -372,4 +372,80 @@ move for some iterations.
 
 # Exercise 3
 
+We first show that the lemma holds for $\tau \in [0,1]$. Since
+
+$$\|\tilde{p}(\tau)\| = \|\tau p^U\| = \tau \|p^U\| \text{ for } \tau \in [0,1]$$
+
+Then the norm of the step $\tilde{p}$ clearly increases monotonically as $\tau$
+increases. For the second criterion, we compute the quadratic model for a
+generic $\tau \in [0,1]$:
+
+$$m(\tilde{p}(\tau)) = f - \frac{\tau^2 \|g\|^2}{g^TBg} - \frac12 \frac{\tau^2
+\|g\|^2}{(g^TBg)^2} g^TBg = f - \frac12 \frac{\tau^2 \|g\|^2}{g^TBg}$$
+
+$g^T B g > 0$ since we assume $B$ is positive definite, therefore the entire
+term in function
+of $\tau^2$ is negative and thus the model for an increasing $\tau \in [0,1]$
+decreases monotonically (to be precise quadratically).
+
+Now we show that the monotonicity claims hold also for $\tau \in [1,2]$. We
+define a function $h(\alpha)$ (where $\alpha = \tau - 1$) with same monotonicity
+as $\|\tilde{p}(\tau)\|$ and we show that this function monotonically increases:
+
+$$h(\alpha) = \frac12 \|\tilde{p}(1 - \alpha)\|^2 = \frac12 \|p^U + \alpha(p^B -
+p^U)\|^2 = \frac12 \|p^U\|^2 + \frac12 \alpha^2 \|p^B - p^U\|^2 + \alpha (p^U)^T
+(p^B - p^U)$$
+
+We now take the derivative of $h(\alpha)$ and we show it is always positive,
+i.e. that $h(\alpha)$ has always positive gradient and thus that is it
+monotonically increasing w.r.t. $\alpha$:
+
+$$h'(\alpha) = \alpha \|p^B - p^U\|^2 + (p^U)^T (p^B - p^U) \geq (p^U)^T (p^B - p^U)
+= \frac{g^Tg}{g^TBg}g^T\left(- \frac{g^Tg}{g^TBg}g + B^{-1}g\right) =$$$$= \|g\|^2
+\frac{g^TB^{-1}g}{g^TBg}\left(1 - \frac{\|g\|^2}{(g^TBg)(g^TB^{-1}g)}\right) $$
+
+Since we know $B$ is symmetric and positive definite, then $B^{-1}$ is as well.
+Therefore, we know that the term outside of the parenthesis is always positive
+or 0. Therefore, we now only need to show that:
+
+$$\frac{\|g\|^2}{(g^TBg)(g^TB^{-1}g)} \leq 1 \Leftrightarrow \|g\|^2 \leq
+(g^TBg)(g^TB^{-1}g)$$
+
+since both factors in the denominator are positive for what we shown before.
+
+We now define a inner product space $\forall a, b \in R^N, \; {\langle a,
+b\rangle}_B = a^T B b$. We now prove that this is indeed a linear product space
+by proving all properties of such space:
+
+- **Linearity w.r.t. the first argument:**
+
+  ${\langle x, y \rangle}_B + {\langle z,
+  y \rangle}_B = x^TBy + z^TBy = (x + z)^TBy = {\langle (x + z), y \rangle}_B$;
+
+- **Symmetry:**
+
+  ${\langle x, y \rangle}_B = x^T B y = (x^T B y)^T = y^TB^Tx$, and
+  since $B$ is symmetric, $y^TB^Tx = y^TBx = {\langle y,x \rangle}_B$;
+
+- **Positive definiteness:**
+
+  ${\langle x, x \rangle_B} = x^T B x > 0$ is true since B is positive definite.
+
+Since ${\langle x, y \rangle}_B$ is indeed a linear product space, then:
+
+$${\langle g, B^{-1} g \rangle}_B \leq {\langle g, g \rangle}_B, {\langle B^{-1}
+g, B^{-1} g \rangle}_B$$
+
+holds according to the Cauchy-Schwartz inequality. Now, if we expand each inner
+product we obtain:
+
+$$g^T B B^{-1} g \leq (g^T B g) (g^T (B^{-1})^T B B^{-1} g)$$
+
+Which, since $B$ is symmetric, in turn is equivalent to writing:
+
+$$g^T g \leq (g^TBg) (g^T B^{-1} g)$$
+
+which is what we needed to show to prove that the first monotonicity constraint
+holds for $\tau \in [1,2]$.
+
 **TBD**