--- header-includes: - \usepackage[utf8]{inputenc} - \usepackage[T1]{fontenc} - \usepackage[sc]{mathpazo} - \usepackage{caption, subcaption} - \usepackage{hyperref} - \usepackage[english]{babel} - \usepackage{amsmath, amsfonts} - \usepackage{listings} - \usepackage{graphicx} - \graphicspath{{Figures/}{./}} - \usepackage{float} - \usepackage{geometry} - \geometry{paper=a4paper,top=2.5cm,bottom=3cm,left=3cm,right=3cm} - \usepackage{sectsty} - \sectionfont{\vspace{6pt}\centering\normalfont\scshape} - \subsectionfont{\normalfont\bfseries} - \subsubsectionfont{\normalfont\itshape} - \paragraphfont{\normalfont\scshape} - \usepackage{scrlayer-scrpage} - \ofoot*{\pagemark} - \ifoot*{Maggioni Claudio} - \cfoot*{} --- \title{ \normalfont\normalsize \textsc{Machine Learning\\ Universit\`a della Svizzera italiana}\\ \vspace{25pt} \rule{\linewidth}{0.5pt}\\ \vspace{20pt} {\huge Assignment 1}\\ \vspace{12pt} \rule{\linewidth}{1pt}\\ \vspace{12pt} } \author{\LARGE Maggioni Claudio} \date{\normalsize\today} \maketitle The assignment is split into two parts: you are asked to solve a regression problem, and answer some questions. You can use all the books, material, and help you need. Bear in mind that the questions you are asked are similar to those you may find in the final exam, and are related to very important and fundamental machine learning concepts. As such, sooner or later you will need to learn them to pass the course. We will give you some feedback afterwards.\ !! Note that this file is just meant as a template for the report, in which we reported **part of** the assignment text for convenience. You must always refer to the text in the README.md file as the assignment requirements. # Regression problem This section should contain a detailed description of how you solved the assignment, including all required statistical analyses of the models' performance and a comparison between the linear regression and the model of your choice. Limit the assignment to 2500 words (formulas, tables, figures, etc., do not count as words) and do not include any code in the report. ## Premise on data splitting In order to perform a correct statistical analysis, I have splitted the given datapoints in a training set, a validation set and a test set. The test set has 200 points (10% of the given datapoints), the validation set has 180 data points (10% of the remaining 90% of the given datapoint) while the training set has 1620 datapoints (all the remaining datapoints). For the linear model, I will perform the linear regression using both the training and the validation datapoints (since there is no need to choose hyperparameters since the given family of models requires none). For the nonlinear model, which is a feedforward NN in my case, I used the training set to fit the model and I have used the validation set in order to choose the architecture of the neural network. ## Task 1 Use the family of models $f(\mathbf{x}, \boldsymbol{\theta}) = \theta_0 + \theta_1 \cdot x_1 + \theta_2 \cdot x_2 + \theta_3 \cdot x_1 \cdot x_2 + \theta_4 \cdot \sin(x_1)$ to fit the data. Write in the report the formula of the model substituting parameters $\theta_0, \ldots, \theta_4$ with the estimates you've found: $$f(\mathbf{x}, \boldsymbol{\theta}) = 3.13696 -0.468921 \cdot x_1 -0.766447 \cdot x_2 + 0.0390513 \cdot x_1 x_2 -0.918346 \cdot \sin(x_1)$$ Evaluate the test performance of your model using the mean squared error as performance measure. On the 200-datapoint test set described before, the MSE of this linear model is $1.22265$. This result can be reproduced by running `src/statistical_tests.py`. ## Task 2 Consider any family of non-linear models of your choice to address the above regression problem. Evaluate the test performance of your model using the mean squared error as performance measure. Compare your model with the linear regression of Task 1. Which one is **statistically** better? The model I've chosen is a feed-forward neural network with 2 hidden layers: - one 22-neuron dense layer with hyperbolic tangent activation function; - one 15-neuron dense layer with sigmoidal activation function; Finally, the output neuron has a linear activation function. As described before, the validation set was used to manually tailor the NNs architecture. The network was trained using the `adam` optimizer with 5000 maximum number of epochs and with an early stopping procedure that has a patience of 120 epochs. The fitted model has these final performance parameters: - Final MSE on validation set $\approx 0.0143344$ - Final MSE on the test set $\approx 0.0107676$ I conclude the NN non-linear model is better than the linear regression model because of the lower test set MSE. Since the test set is 200 elements big, we can safely assume that the loss function applied on the test set is a resonable approximation of the structural risk for both models. In order to motivate this intuitive argument, I will perform a student's T-test to show that the non-linear model is indeed statistically better with 95% confidence. The test was implemented in `src/statistical_tests.py` and it has been performed by spitting the test set, assuming both resulting sets have an expectation of the noise equal to 0, and by computing mean, variance and T-score. The results of the test are the following: - T-test result: 5.37324 - Linear model noise variance: 6.69426 - Feedforward NN noise variance: $0.348799 \cdot 10^{-3}$ The test is clearly outside of the 95% confidence interval of $[-1.96,1.96]$ and thus we reject the null hypothesis. Since the variance is lower for the NN we conclude that this model is indeed the better one statistically speaking. ## Task 3 (Bonus) In the [**Github repository of the course**](https://github.com/marshka/ml-20-21), you will find a trained Scikit-learn model that we built using the same dataset you are given. This baseline model is able to achieve a MSE of **0.0194**, when evaluated on the test set. You will get extra points if the test performance of your model is better (i.e., the MSE is lower) than ours. Of course, you also have to tell us why you think that your model is better. In order to understand if my model is better than the baseline model, I performed another Student's T-test re-using the subset of the test set I've used for my linear model. The results of the new test are the following: - T-test result: 1.1777 (for a $p$ value of 0.2417396) - Baseline model noise variance: $0.366316 \cdot 10^{-3}$ - Feedforward NN noise variance: $0.348799 \cdot 10^{-3}$ This test would accept my model as the better one (as the variance is lower) with a 75% confidence interval, which is far from ideal but still can be significant and could be a good chance at having a better model on the hidden test set. To further analyze the performance of my NN w.r.t. the baseline model, I have computed the MSE over my entire 200-datapoints test set for both models. Here are the results: - MSE on entire test set for baseline model: 0.0151954 - MSE on entire test set for feedforward NN model: 0.0107676 My model shows lower MSE for this particular test set, but even if this test wasn't used to train both sets of course the performance may change for a different test set like the hidden one. The T-test performed before suggests that my model might have a decent chance at performing better than the baseline model, but I cannot say that for sure or with a reasonable (like 90% or 95%) confidence. # Questions ## Q1. Training versus Validation 1. **Explain the curves' behavior in each of the three highlighted sections of the figures, namely (a), (b), and (c).** In the highlighted section (a) the expected test error, the observed validation error and the observed training error are significantly high and close toghether. All the errors decrease as the model complexity increases. In (c), instead, we see a low training error but high validation and expected test error. The last two increase as the model complexity increases while the training error is in a plateau. Finally, in (b), we see the test and validation error curves reaching their respectively lowest points while the training error curve decreases as the model complexity increases, albeit in a less steep fashion as its behaviour in (a). 2. **Is any of the three section associated with the concepts of overfitting and underfitting? If yes, explain it.** Section (a) is associated with underfitting and section (c) is associated with overfitting. The behaviour in (a) is fairly easy to explain: since the model complexity is insufficient to capture the behaviour of the training data, the model is unable to provide accurate predictions and thus all MSEs we observe are rather high. It's worth to point out that the training error curve is quite close to the validation and the test error: this happens since the model is both unable to learn accurately the training data and unable to formulate accurate predictions on the validation and test data. In (c) instead, the model complexity is higher than the intrinsic complexity of the data to model, and thus this extra complexity will learn the intrinsic noise of the data. This is of course not desirable, and the dire consequences of this phenomena can be seen in the significant difference between the observed MSE on training data and MSEs for validation and test data. Since the model learns the noise of the training data, the model will accurately predict noise fluctuations on the training data, but since this noise is completely meaningless information for fitting new datapoints, the model is unable to accurately predict for validation and test datapoints and thus the MSEs for those sets are high. Finally in (b) we observe fairly appropriate fitting. Since the model complexity is at least on the same order of magnitude of the intrinsic complexity of the data the model is able to learn to accurately predict new data without learning noise. Thus, both the validation and the test MSE curves reach their lowest point in this region of the graph. 3. **Is there any evidence of high approximation risk? Why? If yes, in which of the below subfigures?** Depending on the scale and magnitude of the x axis, there could be significant approximation risk. This can be observed in subfigure (b), namely by observing the difference in complexity between the model with lowest validation error and the optimal model (the model with lowest expected test error). The distance between the two lines indicated that the currently chosen family of models (i.e. the currently chosen gray box model function, and not the value of its hyperparameters) is not completely adequate to model the process that generated the data to fit. High approximation risk would cause even a correctly fitted model to have high test error, since the inherent structure behind the chosen family of models would be unable to capture the true behaviour of the data. 4. **Do you think that by further increasing the model complexity you will be able to bring the training error to zero?** Yes, I think so. The model complexity could be increased up to the point where the model would be so complex that it could actually remember all x-y pairs of the training data, thus turning the model function effectively in a one-to-one direct mapping between input and output data of the training set. Then, the loss on the training dataset would be exactly 0. This of course would mean that an absurdly high amount of noise would be learned as well, thus making the model completely useless for prediction of new datapoints. 5. **Do you think that by further increasing the model complexity you will be able to bring the structural risk to zero?** No, I don't think so. In order to achieve zero structural risk we would need to have an infinite training dataset covering the entire input parameter domain. Increasing the model's complexity would actually make the structural risk increase due to overfitting. ## Q2. Linear Regression Comment and compare how the (a.) training error, (b.) test error and (c.) coefficients would change in the following cases: 1. **$x_3$ is a normally distributed independent random variable $x_3 \sim \mathcal{N}(1, 2)$** With this new variable, the coefficients $\theta_1$ and $\theta_2$ will not change significantly for the new optimal model. Training and test error behave similarly, although the training error may be higher in the first iteration of the learning procedure. All this variations are due to the fact that the new variable $x_3$ is completely independent from $x_1$ and $x_2$, and consequently from $y$. Therefore, the model will "understand" that $x_3$ contains no information at all and thus set $\theta_3$ to 0. This effect would be achieved even more quickly by using Lasso instead of linear regression, since Lasso tends to set parameters to zero when their linear regression optimal value would be already close to 0. 1. **$x_3 = 2.5 \cdot x_1 + x_2$** With this new variable, the coefficients would indeed change but test and training error would stay the same. Since $x_3$ is a linear combination of $x_1$ and $x_2$, then we can rewrite the model function in the following way: $$f(x, \theta) = \theta_1 x_1 + \theta_2 x_2 + \theta_3 (2.5 x_1 + x_2) = (\theta_1 + 2.5 \theta_3) x_1 + (\theta_2 + \theta_3) x_2$$ This shows that even if the value of $\theta_1$ and $\theta_2$ would change if this term is introduced, the solution that would be found through linear regression would still be effectively equivalent w.r.t. effectiveness and MSE to the optimal model for the original family of models. 1. **$x_3 = x_1 \cdot x_2$** If the underlying process generating the data would also depend on an $x_1 \cdot x_2$ operation, then this additional input variable would change the parameters, improve the training error, and depending on if the impact of this quadratic term on the original data-generating process is small or big, it would slighty or considerably improve the test error. Essentially, this parameter would had useful complexity to the model, which may be beneficial if the model is underfitted w.r.t. number of variables in the linear regression function, or otherwise detrimental if the model is correctly fitted or overfitted already. ## Q3. Classification 1. **Your boss asked you to solve the problem using a perceptron and now he's upset because you are getting poor results. How would you justify the poor performance of your perceptron classifier to your boss?** The classification problem in the graph, according to the data points shown, is quite similar to the XOR or ex-or problem. Since in 1969 that problem was proved impossible to solve by a perceptron model by Minsky and Papert, then that would be quite a motivation in front of my boss. On a more general (and more serious) note, the perceptron model would be unable to solve the problem in the picture since a perceptron can solve only linearly-separable classification problems, and even through a simple graphical argument we would be unable to find a line able able to separate yellow and purple dots w.r.t. a decent approximation simply due to the way the dots are positioned. 2. **Would you expect to have better luck with a neural network with activation function $h(x) = - x \cdot e^{-2}$ for the hidden units?** First of all we notice that that activation function is still linear, even if a $-e^{-2}$ scaling factor is applied to the output. Since the problem is not linearly separable, we would still have bad luck since the new family of models would still be inadequate for solving the problem. 3. **What are the main differences and similarities between the perceptron and the logistic regression neuron?** The perceptron neuron and the logistic regression neuron both are dense neurons that recieve as parameters all values from the previous layer as single scalars. Both require a number of parameters equal to the number of inputs, which are used as scaling factors for the inputs recieved. The main difference between the neurons is the activation function: the perceptron uses an heavyside activation function, while the logistic regression neuron uses a sigmoidal function, providing a probabliistic interpretation of the output w.r.t. the classification problem (i.e. by instead of providing just a "in X class" or "not in X class" indication, the logistic regression neuron can output in the continuous $[0,1]$ range a probabliity of said element being in the class).