OM/Claudio_Maggioni_4/Claudio_Maggioni_4.md

<!-- vim: set ts=2 sw=2 et tw=80: -->

---
title: Homework 4 -- Optimization Methods
author: Claudio Maggioni
header-includes:
- \usepackage{amsmath}
- \usepackage{hyperref}
- \usepackage[utf8]{inputenc}
- \usepackage[margin=2.5cm]{geometry}
- \usepackage[ruled,vlined]{algorithm2e}
- \usepackage{float}
- \floatplacement{figure}{H}
- \hypersetup{colorlinks=true,linkcolor=blue}

---
\maketitle

# Exercise 1

## Exercise 1.1

The lagrangian is the following:

$$L(X,\lambda) = f(X) - \lambda \left(c(x) - 0\right) = -3x^2 + y^2 + 2x^2 +
2(x+y+z) - \lambda x^2 - \lambda y^2 -\lambda z^2 + \lambda =$$$$= (-3 -\lambda)x^2 + (1-
\lambda)y^2 + (2-\lambda)z^2 + 2 (x+y+z) + \lambda$$

The KKT conditions are the following:

First we have the condition on the partial derivatives of the Lagrangian w.r.t.
$X$:

$$\nabla_X L(X,\lambda) = \begin{bmatrix}(-3-\lambda)x^* + 1\\(1-\lambda)y^* +
1\\(2-\lambda)z^* + 1\end{bmatrix} = 0 \Leftrightarrow
\begin{bmatrix}x^*\\y^*\\z^*\end{bmatrix} =
\begin{bmatrix}\frac1{3+\lambda}\\-\frac1{1-\lambda}\\-\frac{1}{2-\lambda}\end{bmatrix}$$

Then we have the complementarity condition:

$$c(X) = {x^*}^2 + {y^*}^2 + {z^*}^2 - 1 = 0 \Leftrightarrow \|X^*\| = 1$$

$$\lambda^* c(X^*) = 0 \Leftarrow c(X^*) = 0 \text{ which is true if the above
condition is true.}$$

Since we have no inequality constraints, we don't need to apply the KKT
conditions realated to inequality constraints.

## Exercise 1.2

To find feasible solutions to the problem, we apply the KKT conditions. Since we
have a way to derive $X^*$ from $\lambda^*$ thanks to the first KKT condition,
we try to find the values of $\lambda$ that satisfies the second KKT condition:

$$c(x) = \left( \frac{1}{3+\lambda} \right)^2 + \left( -\frac{1}{1-\lambda} \right)^2 +
\left(-\frac{1}{2-\lambda}\right)^2 - 1 =
\frac{1}{(3+\lambda)^2} + \frac{1}{(1-\lambda)^2} + \frac{1}{(2-\lambda)^2} - 1 =$$$$=
\frac{(1-\lambda)^2(2-\lambda)^2 + (3+\lambda)^2(2-\lambda)^2 +
(3+\lambda)^2
(1-\lambda)^2 - (3+\lambda)^2 (1-\lambda)^2 (2-\lambda)^2}{(3+\lambda)^2
(1-\lambda)^2 (2-\lambda)^2} = 0
\Leftrightarrow$$$$\Leftrightarrow
(1-\lambda)^2(2-\lambda)^2 + (3+\lambda)^2(2-\lambda)^2 +
(3+\lambda)^2
(1-\lambda)^2 - (3+\lambda)^2 (1-\lambda)^2 (2-\lambda)^2 = 0
\Leftrightarrow$$$$\Leftrightarrow
(\lambda^4 - 6\lambda^3 + 13\lambda^2 - 12\lambda + 16) +
(\lambda^4 + 2\lambda^3 - 11\lambda^2 - 12\lambda + 36) +
(\lambda^4 + 4\lambda^3 -  2\lambda^2 - 12\lambda + 9)$$$$
+ (\lambda^6 -14\lambda^4 +12\lambda^3 +49\lambda^2 -84\lambda + 36) = $$$$
=-\lambda^6 +17\lambda^4 -12\lambda^3 -49\lambda^2 +48\lambda +13 = 0
\Leftrightarrow $$$$ \Leftrightarrow
\lambda = \lambda_1 \approx -0.224 \lor
\lambda = \lambda_2 \approx -1.892 \lor
\lambda = \lambda_3 \approx 3.149 \lor
\lambda = \lambda_4 \approx -4.035$$

We then compute $X$ from each solution and evaluate the objective each time:

$$X = \begin{bmatrix}\frac1{3+\lambda}\\-\frac1{1-\lambda}\\
-\frac{1}{2-\lambda}\end{bmatrix}
\Leftrightarrow$$$$\Leftrightarrow
X = X_1 \approx \begin{bmatrix}0.360\\-0.817\\-0.450\end{bmatrix} \lor
X = X_2 \approx \begin{bmatrix}0.902\\-0.346\\-0.257\end{bmatrix} \lor
X = X_3 \approx \begin{bmatrix}0.163\\0.465\\0.870\end{bmatrix} \lor
X = X_4 \approx \begin{bmatrix}-0.966\\-0.199\\-0.166\end{bmatrix}$$

$$f(X_1) = -1.1304 \;\; f(X_2) = -1.59219 \;\; f(X_3) = 4.64728 \;\; f(X_4) =
-5.36549$$

## Exercise 1.3

To find the optimal solution, we choose $(\lambda_4, X_4)$ since $f(X_4)$ is the
smallest objective
value out of all the feasible points. Therefore, the solution to the
minimization problem is:

$$X \approx \begin{bmatrix}-0.966\\-0.199\\-0.166\end{bmatrix}$$

# Exercise 2

## Exercise 2.1

To reformulate the problem, we first rewrite the explicit values of $G$, $c$,
$A$ and $b$:

$$G = 2 \cdot \begin{bmatrix}3&0&0\\2&2.5&0\\1&2&2\end{bmatrix}$$
$$c = \begin{bmatrix}-8\\-3\\-3\end{bmatrix}$$
$$A = \begin{bmatrix}1&0&1\\0&1&1\end{bmatrix}$$
$$b = \begin{bmatrix}3\\0\end{bmatrix}$$

Then, using these variable values and the formulation given on the assignment
sheet the problem is restated in this new form.

## Exercise 2.2

The lagrangian for this problem is the following:

$$L(x, \lambda) = \frac12\langle x, Gx\rangle + \langle x, c \rangle - \lambda
(Ax - b) =$$$$= \begin{bmatrix}x_1&x_2&x_3\end{bmatrix}
  \begin{bmatrix}3&0&0\\2&2.5&0\\1&2&2\end{bmatrix}
    \begin{bmatrix}x_1\\x_2\\x_3\end{bmatrix} +
      \begin{bmatrix}x_1&x_2&x_3\end{bmatrix}
        \begin{bmatrix}-8\\-3\\-3\end{bmatrix} - \lambda
          \left(\begin{bmatrix}1&0&1\\0&1&1\end{bmatrix}
            \begin{bmatrix}x_1\\x_2\\x_3\end{bmatrix} -
              \begin{bmatrix}3\\0\end{bmatrix}\right)$$


The KKT conditions are the following:

First we have the condition on the partial derivatives of the Lagrangian w.r.t.
$X$:

$$\nabla_x L(x, \lambda) = Gx + c - A^T \lambda = \begin{bmatrix}3 x_1 - 8 +
  \lambda_1\\ 2x_1 + 2.5 x_2 - 3 + \lambda_2\\x_1 + 2x_2 + 2x_3 - 3 + \lambda_1
+ \lambda_2\end{bmatrix} > 0$$

Then we have the conditions on the equality constraint:

$$Ax - b = 0 \Leftrightarrow \begin{bmatrix}x_1 + x_3\\x_2 + x_3\end{bmatrix} =
  \begin{bmatrix}3\\0\end{bmatrix}$$

Then we have the complementarity condition:

$$\lambda^T (Ax - b) = 0 \Leftarrow Ax - b = 0 \text{ which is true if the above
condition is true.}$$

Since we have no inequality constraints, we don't need to apply the KKT
conditions realated to inequality constraints.

# Exercise 3

## Exercise 3.1

The lagrangian of this problem is the following:

$$L(x, \lambda) = c^T x - \lambda^T (Ax - b) - s^T x$$

The KKT conditions are the following:

1. The partial derivative of the lagrangian w.r.t. $x$ is 0:
   $$\nabla_x L(x, \lambda) = c - A^T \lambda - s = 0 \Leftrightarrow A^T \lambda
   + s = c$$
2. Equality constraints hold:
   $$Ax - b = 0 \Leftrightarrow Ax = b$$
3. Inequality constraints hold:
   $$x \geq 0$$
4. The lagrangian multipliers for inequality constraints are non-negative:
   $$s \geq 0$$
5. The complementarity condition holds (here considering only inequality constraints,
   since the condition trivially holds for equality ones):
   $$s^T x \geq 0$$

## Exercise 3.2

We define the dual problem is the following way:

$$\max b^T \lambda \;\; \text{ s.t. } \;\; A^T \lambda + s = c \; \text{ and
}\; s \geq 0$$

To convert this maximization problem in a minimization one, we flip the sign of
the objective and we find:

$$\min - b^T \lambda \;\; \text{ s.t. } \;\; A^T \lambda + s = c \; \text{ and
}\; s \geq 0$$

We then compute the Lagrangian of the dual problem:

$$L(\lambda, x) = -b^T \lambda + x^T (A^T \lambda + s - c) - x^T s  = - b^T \lambda + x^T (A^T \lambda - c)$$

The KKT conditions are the following:

1. The partial derivative of the lagrangian w.r.t. $x$ is 0:
   $$\nabla_{\lambda} L(\lambda, x) = - b^T + x^T A^T = 0 \Leftrightarrow Ax = b$$
2. Equality constraints hold:
   $$A^T \lambda + s = c$$
3. Inequality constraints hold:
   $$c - A^T \lambda \geq 0 \Leftrightarrow s \geq 0 \;\;\; \text{ using 2.\ to find
   that } s = c - A^T \lambda$$
4. The lagrangian multipliers for inequality constraints are non-negative:
   $$x \geq 0$$
5. The complementarity condition holds (here considering only inequality constraints,
   since the condition trivially holds for equality ones):
   $$x^T s \geq 0 \Leftrightarrow s^T x \geq 0$$

Then, if we compare the KKT conditions of the primal problem with the ones above
we can match them to see that they are identical:

- 1.\ from the dual is identical to 2.\ from the primal;
- 2.\ from the dual is identical to 1.\ from the primal;
- 3.\ from the dual is identical to 4.\ from the primal;
- 4.\ from the dual is identical to 3.\ from the primal;
- 5.\ from the dual is identical to 5.\ from the primal.

Therefore, the primal and the dual problem are equivalent.