Local Regression

We slide a window across the predictor values, and only observations in that window are used to make a regression model.

The model here could be a linear regression, or even a polynomial model. After all, we average the values as they go further.

• 🤬 Hard to write up an equation or function easily.

• 😀 Does well in the awkward relationship.

1. Take a sliding window to compute regression model

2. then combine the results by averaging

Assumption:

• Around point x, the mean of y can be approximated by a small class of parametric functions in polynomial regression.

• The errors in estimating y are independent and randomly distributed with a mean of zero.

• Bias and variance are traded off by the choices for the settings of span and degree of polynomial.

Polynomial Regression

Polynomial Regression basically transforms (e.g., making x to be cubic) the variable to make a new variable, and then fit a model

Orthogonal Polynomial Regression makes the variables to be as uncorrelated as possible with the first variable; there's no correlation between the two variables so it sets up a system of new variables that are really just polynomials, but that's a special polynomial it adds additional parts

Assumption:

• the behavior of a dependent variable y is explained by a linear, or nonlinear — curvilinear, additive relationship between the dependent variable and a set of k independent variables (xi, i=1 to k)),

• the relationship between the dependent variable y and any independent variable xi is linear or curvilinear (specifically polynomial),

• the independent variables xi are independent of each other

• the errors are independent, normally distributed with mean zero and a constant variance (OLS).

Step Functions (Skipped in lecture 2 but used in tree models)

Basis Functions

Regression Splines & Smoothing Splines

• For each sample, the R2 increases as deg increases.

• The fitted curve is more flexible as deg increases. Deg 10 appears to capture the true model shape better.

Generalized Additive Models

Assumption (Further here)

FAQ

Reference

Extra Resource