Statistical Machine Learning (Summer term 2020)

Course material

Slides (publically available): Latest version, updated 2020-08-19: pdf
Videos (Publically available): The videos of the lecture can all be found on youtube.

Assignments (only accessible for students who are enrolled in the course):
Recap material:

Background information about the course

Lecture

The lecture is intended for Master students in computer science or mathematics, but might also be interesting for students of other disciplines like physics, linguistics, economics, etc. If in doubt, please simply watch the first lectures and talk to us during the Questions and Answers session. Lectures start in the week of April 20 and do not require your presence. Instead, they will be recorded videos that everybody can watch at home.


Lectures (by Prof. U. von Luxburg). In case we will need them later this term, the time slots and booked rooms are Tuesday 8:15 - 10:00, Hörsaal 036, Neuphilologicum, Wilhelmstraße 50; Thursday 8:15-10:00, Hörsaal 22, Kupferbau, Hölderlinstraße 5.

Tutorials

Tutorials will be held online via zoom. Tutorials do require your online presence and will not be recorded. In the first week of of April 20, there will be no tutorials. In the second week of April 27, there will be only one tutorial session. It will take place on Wednesday April 29 from 16-18. This tutorial will give an introduction to Python. You should attend this tutorial if you are not already familiar with Python and Numpy. From the third week of May 4, there will be multiple tutorial sessions every week. You can choose between Tuesday 14-16 (group 1), Tuesday 16-18 (group 2), Wednesday 14-16 (group 3), Wednesday 16-18 (group 4) and Thursday 16-18 (group 5). The allocation to the tutorials is available here.

As usual, you will hand in a weekly assignment, both code and math. You will send everything to us via email and get your correxted assignments, both code and math, returned to you via email. Math and coding exercises will then be discussed in the tutorials. We encourage you to work in groups to solve the exercises. To hand in the exercises, please form groups of two students (that is, two students jointly hand in solutions). Note that both students need to be familiar with all the solutions their group submits, so they can present them in the tutorial sessions (even if they are online!). UPDATE: Every student is required to submit with a partner.

In addition to the tutorials, tutors are going to offer Question and Answer sessions during the first weeks of the semester. There you can ask organizational questions regarding the lecture, tutorials and assignments. The second session will take place on Tuesday April 28, 16:00-16:30, via zoom.

Teaching assistants and their contacts:

Contents of the lecture

The focus of the lecture is on both algorithmic and theoretic aspects of machine learning. We will cover many of the standard algorithms and learn about the general principles and theoretic results for building good machine learning algorithms. Topics range from well-established results to very recent results.

The basics:
  • Bayesian decision theory, no free lunch theorem.
  • Supervised learning problems (regression, classification): Simple baselines (nearest neibhbor, random forests); linear methods; regularization; non-linear kernel methods and the theory behind it
  • Unsupervised learning problems: Dimension reduction from PCA to manifold methods; Clustering from k-means to spectral clustering and spectral graph theory
  • Statistical Learning Theory: consistency and generalization bounds
  • How to model machine learning problems
Advanced topics:
  • Low rank matrix completion, compressed sensing
  • Ranking
  • Fairness in machine learning
  • Online learning and its theory
The following topics will NOT be covered, as there are alternative, dedicated lectures to each of these topics in our department: neural networks and deep learning; Bayesian and probabilistic machine learning; reinforcement learning.

Prerequisites

You need to know the basics of probability theory and linear algebra, as taught in the mathematics for computer science lectures in your bachelor degree, or even better as taught in the class Mathematics for Machine Learning. If you cannot remember them so well, I strongly urge you to recap the material (see below for links).

Assessment criteria and exams

There will be weekly assignments that you have to solve and hand in: theoretical assignments and programming assignments in python (an introduction to python will be given in one of the first tutorials; it will not be possible to use any other programming language). You need to achieve at least 50% of all assignment points to be admitted to the final exam.

Admissions of previous years: if you have attended the course Machine learning: algorithms and theory in 2019, and you got admitted to the exam, you do not need to get a new admission (we strongly encourage you to take part in the exercises, but you do not have to). Any admission before 2019 is invalid, you will need to the assignments again.

The final exams are written (Klausur).

You are not allowed to bring any material (books, slides, etc) except for what we call the controlled cheat sheet: one side (A4, one side only) of handwritten (!) notes, made by yourself.

Literature

There is no textbook that covers machine learning the way I want. We are going to use individual chapters of various books. Here is a list of some relevant books, some of which also exist as online versions (for the rest, please check out the library):
  • Shalev-Shwartz, Ben-David: Understanding Machine Learning. 2014.
  • Chris Bishop: Pattern recognition and Machine Learning. 2006.
  • Hastie, Tibshirani, Friedman: Elements of statistical learning. 2001. (the statistics point of view on machine learning, written by statisticians)
  • Kevin Murphy: Machine Learning, a probabilistic perspective, 2012 (for the probabilistic point of view)
  • Schölkopf, Smola: Learning with kernels. 2002 (for support vector machines and kernel algorithms)
  • Hastie, Tibshirani, Wainwright: Statistical learning with sparsity. 2015. (Lasso, low rank matrix methods)
  • Duda, Hart, Stork: Pattern classification. (a somewhat outdated classic, but some sections still good to read)
  • Mohri, Rostamizadeh, Talwalkar: Foundations of Machine Learning. 2012.(rather theoretical point of view)
  • Devroye, Györfi, Lugosi: A Probabilistic Theory of Pattern Recognition. 2013. (Covers statistical learning theory, we do not use much of it in this lecture.)