BYNotes

Statistical Learning Algorithms

2 min read

Statistical learning algorithm notes covering linear models, classification algorithms, nonparametric methods, ensemble learning, support vector machines, and kernel methods.

Start Here

For learning statistical learning algorithms, read the modules in this order:

  1. Probability prerequisites: Limit Theorems and Convergence and Distribution Theory.
  2. Computational checks: Computational Statistics.
  3. Linear Models and Regression.
  4. Classification Algorithms.
  5. Nonparametric and Flexible Models.
  6. Ensemble Learning.
  7. Kernel Methods.

Prerequisite Map

NeedProbability / inference backgroundAlgorithm modules
Estimator behavior and approximationLimit Theorems and ConvergenceLinear Models and Regression, Classification Algorithms
Likelihood and probabilistic modelsDistribution Theory, Statistical InferenceClassification Algorithms, Linear Models and Regression
Simulation and sanity checksComputational StatisticsNonparametric and Flexible Models, Ensemble Learning

Module Map

ModuleScope
Linear Models and RegressionOLS, locally weighted regression, regularization, and GLMs
Classification AlgorithmsLinear classifiers, generative classifiers, and margin methods
Nonparametric and Flexible ModelsTrees, step functions, regression splines, and smoothing splines
Ensemble LearningBagging, boosting, AdaBoost, and gradient boosting
Kernel MethodsKernels, RKHS, representer theorem, and kernelized learning

Notes

Linear Models and Regression

AreaNoteStatus
RegressionLinear Regressiondraft
RegressionLocally Weighted Linear Regressiondraft
RegressionRegularizationdraft
GLMGeneralized Linear Modelsdraft

Classification Algorithms

AreaNoteStatus
Linear classificationLinear Models for Classificationdraft
Generative modelNaive Bayes Classifierdraft
Generative modelGaussian Discriminant Analysisdraft
Margin methodSupport Vector Machinedraft

Nonparametric and Flexible Models

AreaNoteStatus
Tree modelTree-Based Modelsdraft
Basis expansionStep Functionsdraft
Basis expansionRegression Splinesdraft
Basis expansionSmoothing Splinesdraft

Ensemble Learning

AreaNoteStatus
OverviewEnsemble Learningdraft
AggregationBagging and Pastingdraft
BoostingBoostingdraft
BoostingAdaBoostdraft
BoostingGradient Boostingdraft

Kernel Methods

AreaNoteStatus
Kernel methodsKernel Methodsdraft
RKHSRKHS and Representer Theoremdraft

Concept Map

  • Linear prediction and regularization: OLS, locally weighted regression, ridge, LASSO, elastic net, and GLMs.
  • Classification: logistic regression, perceptron, GDA, Naive Bayes, and SVM.
  • Flexible regression: tree models, step functions, regression splines, and smoothing splines.
  • Ensembles: voting, bagging, boosting, AdaBoost, and gradient boosting.
  • Kernel methods: feature maps, kernel trick, RKHS, representer theorem, and kernelized regression.

Planned Notes

  • General Linear Models
  • Transfer Learning
  • Introduction to Meta Learning
  • Introduction to Reinforcement Learning
  • LifeLong Learning
  • Optimization Algorithms
  • Automatically Determining Hyperparameters
  • Conformal Prediction
  • k-NN
  • Random Forest

5 items under this folder.