hw5: done (check 1.3 antiplagiarism)

Claudio_Maggioni_5/Claudio_Maggioni_5.md

@@ -5,6 +5,7 @@ title: Homework 5 -- Optimization Methods
 author: Claudio Maggioni
 header-includes:
 - \usepackage{amsmath}
+- \usepackage{amssymb}
 - \usepackage{hyperref}
 - \usepackage[utf8]{inputenc}
 - \usepackage[margin=2.5cm]{geometry}
@@ -57,13 +58,76 @@ objective function with is the summation of:
   point $x$ violates that constraint and zero otherwise.
 
 With some fine tuning of the coefficients for these new "penalty" terms, it is
-possible to build an equivalend unconstrained minimization problem whose
+possible to build an equivalent unconstrained minimization problem whose
 minimizer is also constrained minimizer for the original problem.
 
 ## Exercise 1.2
 
+The simplex method, as said in the previous section, works by iterating along
+basic feasible points and minimizing the cost function along them. In terms of
+strict linear algebra terms, the simplex method works by finding an initial set
+of indices $\mathcal{B}$ which represent column indices of $A$. At each
+iteration, the lagrangian inequality set of constraints $s$ is computed and
+checked for negative components, since in order to satisfy the KKT conditions
+the components must be all $\geq 0$. The iteration consists of removing one of
+said components by changing the $\mathcal{B}$ index set, by effectively swapping
+one of the basis vectors with one of the non-basic ones. The method terminates
+once all components of $s$ are non-negative.
+
+Geometrically speaking the meaning of each iteration is a simple transition from
+a basic feasible point to another neighboring one, and the search step is
+effectively equivalent to one of the edges of the polytope. If the $s$ component
+is always chosen to be the smallest (i.e. we choose the "most negative"
+component), then the method behaves effectively a gradient descent operation of
+the cost function hopping between basic feasible points.
+
 ## Exercise 1.3
 
+In order to discuss in detail the interior point method, we first define two
+sets named "feasible set" ($\mathcal{F}$) and "strictly feasible set"
+($\mathcal{F}^{o}$) respectively:
+
+$$
+\begin{array}{l} \mathcal{F}=\left\{(x, \lambda, s) \mid A x=b, A^{T}
+\lambda+s=c,(x, s) \geq 0\right\} \\ \mathcal{F}^{o}=\left\{(x, \lambda, s) \mid
+A x=b, A^{T} \lambda+s=c,(x, s)>0\right\} \end{array}
+$$
+
+A central path $\mathcal{C}$ is defined as an arc strictly composed of feasible
+points which is parametrized by a scalar $\tau>0$ where each point
+$\left(x_{\tau}, \lambda_{\tau}, s_{\tau}\right) \in \mathcal{C}$ satisfies the
+following conditions:
+
+$$
+\begin{array}{c}
+A^{T} \lambda+s=c \\
+A x=b \\
+x_{i} s_{i}=\tau \quad i=1,2, \ldots, n \\
+(x, s)>0
+\end{array}
+$$
+
+We can observe how these conditions are very much similar to the KKT conditions
+and, in fact, they only differ by the factor $\tau$ and by requiring the
+pairwise products to be strictly greater than zero. With this, we can define the
+central path as follows:
+
+$$
+\mathcal{C}=\left\{\left(x_{\tau}, \lambda_{\tau}, s_{\tau}\right) \mid
+\tau>0\right\}
+$$
+
+Given these definitions, we can also observe that, as $\tau$ approaches zero,
+the conditions we have defined become a closer and closer approximation to the
+original KKT conditions. Therefore, if the central path $\mathcal{C}$ converges
+to anything as $\tau$ approaches zero, then we know that it will converge to a
+solution of the linear program. Meaning that the central path is leading us to a
+solution by maintaining $x$ and $s$ positive while reducing the pairwise
+products to zero at the same time. Usually, the Newton method is used to take
+steps following $\mathcal{C}$ rather than by following the set of feasible
+points $\mathcal{F}$ because it allows for longer steps before violating the
+positivity constraint.
+
 # Exercise 2
 
 ## Exercise 2.1