Regularization/Coefficients Shrinkage

Motivation

Metrics to evaluate a Regression Model

• We have so far discussed the p-value from the F-test of an ANOVA
• What about R2 and Adj R2
• For R2 (It is nothing but SSM/SST)

            → From the ANOVA output SST, SSM/Regression, SSE/Residual

•  Adj R2 is nothing but: 1-[(1-R2)(n-1)/n-k-1]

Regularization Techniques

Going beyond variable selection, what about variable shrinkage

• Multicollinearity and the potential for many forms of a regression equation
• Y = 4A-2B  or  Y = 10A-8B

Ridge Regression 

Lasso Regression