DSEG 635: Learning From Data

Prof. Abdelkader Baggag

Hamad Bin Khalifa University (HBKU), Qatar

  • (+974) 4454-7250

  • abaggag [at] hbku.edu.qa

  • Doha, Qatar

Description

This course covers the theory, algorithms, and applications of computational learning. The technical topics covered include linear models, theory of generalization, regularization and validation, neural networks, support vector machines, as well as specialized techniques and a term-long project with big datasets.

  • Prerequisites

    • ICT-605 Applied Data Analytics or DSEG-560 Machine Learning.
    • Ability to program and develop algorithms in some programming language
  • Textbook

    • Learning From Data by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, Hsuan-Tien Lin.

Content

THEORY

1 The Learning Problem: what is learning? is learning feasible?
2 Training versus Testing: can we learn? dichotomies; growth function; break point.
3 Theory of Generalization: the Vapnick-Chervonenkis inequality.
4 The VC Dimension and Learning: scope of VC analysis; utility of VC dimension; generalization bounds.
5 Bias-Variance Tradeoff: bias and variance; learning curves.

TECHNIQUES (MODELS & METHODS)

1 The Linear Model I: linear classification; linear regression; nonlinear transformation.
2 The Linear Model II: logistic regression; likelihood measure; gradient descent.
3 Maximum Likelihood: design and implementation of a max-likelihood solution beyond logistic regression, e.g., Cox regression.
4 Neural Networks: multilayer perceptrons; implementation of a simple neural network from scratch using backpropagation algorithm and stochastic gradient descent, without packages.
5 Overfitting & Regularization: constraining the model; weight decay; augmented error; how does regularization work; implementation of LASSO solver using Coordinate Descent.
6 Support Vector Machines: maximizing the margin; support vectors; nonlinear transforms; implementation of an SVM solver using the algorithm Coordinate Descent and SMO.
7 Kernel Methods: the kernel trick; soft-margin SVM.
8 Radial Basis Functions: RBF and nearest neighbors; RBF and neural networks; RBF and regularization.
9 Convex Optimization: Newton’s method, Nesterov accelerated gradient.
10 Constrained Optimization: augmented Lagrangian multipliers, KKT conditions.
11 A Peak at Unsupervised Learning: k-means clustering; Probability Density Estimation; Gaussian Mixture Models.

PARADIGMS

1 Supervised Learning
2 Unsupervised Learning
3 Reinforcement Learning
4 Active Learning
5 Online Learning
6 Bayesian Learning
7 Graphical Models

Resources

Machine Learning is a subject with a lot of very good expertise and tutorials. It is best to tap on these resources, as they have good production quality and are more condensed. However, we still recommend in-class lectures as they are helpful in building better connection with the materials. Students are highly encouraged to listen to The Video Lectures of Prof. Yaser Abu-Mostafa.