RSM318H1 Lecture Notes - Lecture 5: Tikhonov Regularization, Function Approximation, Kernel Method
Document Summary
Goal is to use some inputs to predict values of outputs. Can be quantitative output (regression) or qualitative output (classification) In machine learning, we are interested in out-of-sample prediction. Nk(x) is neighborhood of x defined by k closest points xi in training sample. Y is average value based on average of k-closest y-values for the closest x-values that are observed. More volatile in terms of expected prediction error. Prediction performance of least squares estimates can be poor. Often has low bias, large variance, especially if p is large. We often want smaller subset that exhibits strongest effects. Don"t know which x to use to predict y. For each , find best subset of x that minimizes rss. Best subset curve, rss, is decreasing in k. Choose k to minimize estimate of expected prediction error. Choosing optimal k: k-fold cross-validation on i-th part. Use other k-1 parts as training data to fit model, calculate prediction error of fitted model.
