hw4: GSPPN-DUMBIFIED

Claudio_Maggioni_4/Claudio_Maggioni_4.md

@@ -93,7 +93,7 @@ $$f(X_1) = -1.1304 \;\; f(X_2) = -1.59219 \;\; f(X_3) = 4.64728 \;\; f(X_4) =
 To find the optimal solution, we choose $(\lambda_4, X_4)$ since $f(X_4)$ is the
 smallest objective
 value out of all the feasible points. Therefore, the solution to the
-minimization problem is:
+constrained minimization problem is:
 
 $$X \approx \begin{bmatrix}-0.966\\-0.199\\-0.166\end{bmatrix}$$
 
@@ -102,15 +102,70 @@ $$X \approx \begin{bmatrix}-0.966\\-0.199\\-0.166\end{bmatrix}$$
 ## Exercise 2.1
 
 To reformulate the problem, we first rewrite the explicit values of $G$, $c$,
-$A$ and $b$:
+$A$ and $b$.
 
-$$G = 2 \cdot \begin{bmatrix}3&0&0\\2&2.5&0\\1&2&2\end{bmatrix}$$
+We first define matrix $G$ as a set of 9 unknown variables and $c$ a set of 3
-$$c = \begin{bmatrix}-8\\-3\\-3\end{bmatrix}$$
+unknown variables:
-$$A = \begin{bmatrix}1&0&1\\0&1&1\end{bmatrix}$$
-$$b = \begin{bmatrix}3\\0\end{bmatrix}$$
 
-Then, using these variable values and the formulation given on the assignment
+$$G = \begin{bmatrix}g_{11}&g_{12}&g_{13}\\g_{21}&g_{22}&g_{23}\\
-sheet the problem is restated in this new form.
+g_{31}&g_{32}&g_{33}\end{bmatrix} \;\;\; c =
+\begin{bmatrix}c_1\\c_2\\c_3\end{bmatrix}$$
+
+We then define $f(x)$ in the following way:
+
+$$f(x) = \frac12 \cdot \begin{bmatrix}x_1&x_2&x_3\end{bmatrix}
+\begin{bmatrix}g_{11}&g_{12}&g_{13}\\g_{21}&g_{22}&g_{23}\\
+g_{31}&g_{32}&g_{33}\end{bmatrix} \begin{bmatrix}x_1\\x_2\\x_3\end{bmatrix}
+- \begin{bmatrix}c_1&c_2&c_3\end{bmatrix}\begin{bmatrix}x_1\\x_2\\x_3\end{bmatrix}
+=$$$$=x_1^2 \cdot \frac{g_{11}}{2} + x_2^2 \cdot \frac{g_{22}}{2} +
+x_3^2 \cdot \frac{g_{33}}{2} +
+\left(\frac{g_{12} + g_{21}}{2}\right) x_1 x_2
++ \left(\frac{g_{13} + g_{31}}{2}\right) x_1 x_3 +
+\left(\frac{g_{23} + g_{32}}{2}\right) x_2 x_3 + c_1 x_1 + c_2 x_2 + c_3 x_3$$
+
+Then, we equal this polynomial to the given one, finding the following values
+and constraints for the coefficients of $G$ and $g$:
+
+$$\begin{cases}g_{11} = 3 \cdot 2 = 6 \\
+g_{22} = 2.5\cdot 2 = 5 \\
+g_{33} = 2 \cdot 2 = 4 \\
+c_1 = -8 \\
+c_2 = -3 \\
+c_3 = -3 \\
+g_{13} + g_{31} = 1 \cdot 2 = 2 \\
+g_{12} + g_{21} = 2 \cdot 2 = 4 \\
+g_{23} + g_{32} = 2 \cdot 2 = 4 \end{cases}$$
+
+As it can be seen by the system of equations above, we have infinite possibility
+for choosing the components of the $G$ matrix that are not on the main diagonal.
+Due to personal taste, we choose those components in such a way that the
+resulting $G$ matrix is symmetric. We therefore obtain:
+
+$$G = \begin{bmatrix}6&2&1\\2&5&2\\1&2&4\end{bmatrix} \;\;\;
+c = \begin{bmatrix}-8\\-3\\-3\end{bmatrix}$$
+
+We perform a similar process for matrix $A$ and vector $b$
+
+$$Ax = b \Leftrightarrow
+\begin{bmatrix}a_{11}&a_{12}&a_{13}\\a_{21}&a_{22}&a_{23}\end{bmatrix}
+\begin{bmatrix}x_1\\x_2\\x_3\end{bmatrix} =
+\begin{bmatrix}b_1\\b_2\end{bmatrix} \Leftrightarrow
+\begin{cases}a_{11}x_1 + a_{12}x_2 + a_{13}x_3 = b_1\\a_{21}x_1 + a_{22}x_2 +
+a_{23}x_3 = b_2\end{cases}$$
+
+To make this system match the given system of equality constraints, we need to
+set the components of $A$ and $b$ in the following way:
+
+$$\begin{cases}a_{11} = 1\\a_{12} = 0\\a_{13} = 1\\a_{21} =
+0\\a_{22}=1\\a_{23}=1 \\b_1 = 3 \\b_2 = 0\end{cases}$$
+
+Therefore, we obtain the following $A$ matrix and $b$ vector:
+
+$$A = \begin{bmatrix}1&0&1\\0&1&1\end{bmatrix} \;\;\; b = \begin{bmatrix}3\\0\end{bmatrix}$$
+
+Then, using these $G$, $c$, $A$ and $b$ values, and using the quadratic
+formulation of the problem written on the assignment
+sheet, the problem is restated in the desired new form.
 
 ## Exercise 2.2
 
@@ -118,7 +173,7 @@ The lagrangian for this problem is the following:
 
 $$L(x, \lambda) = \frac12\langle x, Gx\rangle + \langle x, c \rangle - \lambda
 (Ax - b) =$$$$= \begin{bmatrix}x_1&x_2&x_3\end{bmatrix}
-  \begin{bmatrix}3&0&0\\2&2.5&0\\1&2&2\end{bmatrix}
+  \begin{bmatrix}6&2&1\\2&5&2\\1&2&4\end{bmatrix}
     \begin{bmatrix}x_1\\x_2\\x_3\end{bmatrix} +
       \begin{bmatrix}x_1&x_2&x_3\end{bmatrix}
         \begin{bmatrix}-8\\-3\\-3\end{bmatrix} - \lambda
@@ -132,9 +187,11 @@ The KKT conditions are the following:
 First we have the condition on the partial derivatives of the Lagrangian w.r.t.
 $X$:
 
-$$\nabla_x L(x, \lambda) = Gx + c - A^T \lambda = \begin{bmatrix}3 x_1 - 8 +
+$$\nabla_x L(x, \lambda) = Gx + c - A^T \lambda =
-  \lambda_1\\ 2x_1 + 2.5 x_2 - 3 + \lambda_2\\x_1 + 2x_2 + 2x_3 - 3 + \lambda_1
+\begin{bmatrix}
-+ \lambda_2\end{bmatrix} > 0$$
+6 x_1 + 2 x_2 +   x_3 - 8 + \lambda_1\\
+2 x_1 + 5 x_2 + 2 x_3 - 3 + \lambda_2\\
+1 x_1 + 2 x_2 + 4 x_3 - 3 + \lambda_1 + \lambda_2\end{bmatrix} = 0$$
 
 Then we have the conditions on the equality constraint:
 
@@ -168,18 +225,24 @@ The KKT conditions are the following:
    $$x \geq 0$$
 4. The lagrangian multipliers for inequality constraints are non-negative:
    $$s \geq 0$$
-5. The complementarity condition holds (here considering only inequality constraints,
+5. The complementarity condition holds (here considering only inequality
-   since the condition trivially holds for equality ones):
+   constraints, since the condition trivially holds for equality ones): $$s^T x
-   $$s^T x \geq 0$$
+   \geq 0$$
 
 ## Exercise 3.2
 
 We define the dual problem is the following way:
 
+$$\max b^T \lambda \;\; \text{ s.t. } \;\; A^T \lambda \leq c \;$$
+
+We then introduce a slack variable $s$ to find the equality and inequality
+constraints:
+
 $$\max b^T \lambda \;\; \text{ s.t. } \;\; A^T \lambda + s = c \; \text{ and
 }\; s \geq 0$$
 
-To convert this maximization problem in a minimization one, we flip the sign of
+To convert this maximization problem in a minimization one (in order to achieve
+standard form), we flip the sign of
 the objective and we find:
 
 $$\min - b^T \lambda \;\; \text{ s.t. } \;\; A^T \lambda + s = c \; \text{ and
@@ -187,22 +250,21 @@ $$\min - b^T \lambda \;\; \text{ s.t. } \;\; A^T \lambda + s = c \; \text{ and
 
 We then compute the Lagrangian of the dual problem:
 
-$$L(\lambda, x) = -b^T \lambda + x^T (A^T \lambda + s - c) - x^T s  = - b^T \lambda + x^T (A^T \lambda - c)$$
+$$L(\lambda, x) = -b^T \lambda + x^T (A^T \lambda + s - c) - x^T s  = - b^T
+\lambda + x^T (A^T \lambda - c)$$
 
 The KKT conditions are the following:
 
-1. The partial derivative of the lagrangian w.r.t. $x$ is 0:
+1. The partial derivative of the lagrangian w.r.t. $x$ is 0: $$\nabla_{\lambda}
-   $$\nabla_{\lambda} L(\lambda, x) = - b^T + x^T A^T = 0 \Leftrightarrow Ax = b$$
+   L(\lambda, x) = - b^T + x^T A^T = 0 \Leftrightarrow Ax = b$$
-2. Equality constraints hold:
+2. Equality constraints hold: $$A^T \lambda + s = c$$
-   $$A^T \lambda + s = c$$
+3. Inequality constraints hold: $$c - A^T \lambda \geq 0 \Leftrightarrow s \geq
-3. Inequality constraints hold:
+   0 \;\;\; \text{ using 2.\ to find that } s = c - A^T \lambda$$
-   $$c - A^T \lambda \geq 0 \Leftrightarrow s \geq 0 \;\;\; \text{ using 2.\ to find
+4. The lagrangian multipliers for inequality constraints are non-negative: $$x
-   that } s = c - A^T \lambda$$
+   \geq 0$$
-4. The lagrangian multipliers for inequality constraints are non-negative:
+5. The complementarity condition holds (here considering only inequality
-   $$x \geq 0$$
+   constraints, since the condition trivially holds for equality ones): $$x^T s
-5. The complementarity condition holds (here considering only inequality constraints,
+   \geq 0 \Leftrightarrow s^T x \geq 0$$
-   since the condition trivially holds for equality ones):
-   $$x^T s \geq 0 \Leftrightarrow s^T x \geq 0$$
 
 Then, if we compare the KKT conditions of the primal problem with the ones above
 we can match them to see that they are identical: