RSM412H1 Lecture Notes - Lecture 9: Overfitting, Variance Reduction, Gradient Boosting
Document Summary
Combine multiple models together to produce a more powerful model. Combine several machine learning techniques into one predictive model in order to: a. High variance comes from using too much information from the data. Relies on information that is only relevant for the training set. Model will change a lot depending on the training set b. Bias means model is not able to fully use information in data. Most frequent case, mean response, a few powerful features. Can come from making simplifying assumptions that are wrong. Previous assumptions are incorporated into the model rather than relying solely on data. All parametric models have bias because parameters are assumptions: bias decreases with more complex models. But more complex models will suffer from overfitting c. Most of the time, single base learning algorithm is used. We have homogenous weak learners that are trained in different ways. Can also use different type of base learning algorithms.
