Advanced Topics in Machine Learning: Kernel Methods

Arthur Gretton (with Zoltan Szabo, Kacper Chwialkowski), University College London


All lecture locations are listed on p. 4 of the first set of slides.

Course announcements will be posted on the mailing list.

This page will contain slides and detailed notes for the kernel part of the course. The assignment may also be found here (at the bottom of the page). Note that the slides will be updated as the course progresses, and I modify them to answer questions I get in the classes. I'll put the date of last update next to each document - be sure to get the latest one. Let me know if you find errors.

There are sets of practice exercises and solutions further down the page (after the slides).

See David Silver's page for the reinforcement learning part of the course.

Slides and notes

Lectures 1, 2, and 3 slides and notes, last modified 26 Jan 2016
  • Definition of a kernel, how it relates to a feature space
  • Combining kernels to make new kernels
  • The reproducing kernel Hilbert space
  • Applications: difference in means, kernel PCA, kernel ridge regression

Lectures 4, 5, 6, and 7 slides and notes, last modified 23 Feb 2016

  • Distance between means in RKHS, integral probability metrics, the maximum mean discrepancy (MMD), two-sample tests
  • Choice of kernels for distinguishing distributions, characteristic kernels
  • Covariance operator in RKHS: proof of existence, definition of norms (including HSIC, the Hilbert-Schmidt independence criterion)
  • Application of HSIC to independence testing
  • Application of HSIC to feature selection, taxonomy discovery.
  • Introduction to independent component analysis, kernel ICA

Lecture 8 slides and notes, last modified 26 Jan 2016

  • Introduction to convex optimization
  • The representer theorem
  • Large margin classification, support vector machines for clasification

Lecture 9 slides, lecture 10 slides , and notes, last modified 20 Mar 2013

  • Metric, normed, and unitary spaces, Cauchy sequences and completion, Banach and Hilbert spaces
  • Bounded linear operators and the Riesz Theorem
  • Equivalent notions of an RKHS: existence of reproducing kernel, boundedness of the evaluation operator
  • Positive definiteness of reproducing kernels, the Moore-Aronszajn Theorem
  • Mercer's Theorem for representing kernels

Supplementary lecture slides, last modified 22 Mar 2012

  • Loss and risk, estimation and approximation error, a new interpretation of MMD
  • Why use an RKHS: comparison with other function classes (Lipschitz and bounded Lipschitz)
  • Characteristic kernels and universal kernels


The assignment (first part due in on Thursday March 24th 2016). You will need this extract on incomplete Cholesky (scanned from Shawe-Taylor and Cristianini, Kernel Methods for Pattern Analysis). Last modified 05 Jan 2016.

Practice exercises and solutions

The exercises are taken from exams in previous years, with minor modifications. Worked solutions are provided. Last modified 18 Oct 2015.
  • Set 1
  • Set 2
  • Dates:
    • Free schedule
    Course properties:
    • Free:
    • Paid:
    • Certificate:
    • MOOC:
    • Video:
    • Audio:
    • Email-course:
    • Language: English Gb


    No reviews yet. Want to be the first?

    Register to leave a review

    Included in selections:
    Small-icon.hover Deep Learning
    Good materials on deep learning.
    Small-icon.hover Machine Learning
    Machine learning: from the basics to advanced topics. Includes statistics...
    More from 'Computer Science':
    Acea3daf-ca58-4998-b08c-ba76b97ce78a-1f1dd684c645.small iLabX – The Internet Masterclass
    You want to know how the Internet works? You want to fully understand its mechanisms...
    C2750912-8e29-426f-91b8-c03b0dd9ee8f-d3ce8d3f0f02.small Autonomous Mobile Robots
    Basic concepts and algorithms for locomotion, perception, and intelligent navigation...
    776db6bd-54a0-4625-ba3d-1204fb922859-1df9ac41ffdf.small HTML5 Coding Essentials and Best Practices
    Learn how to write Web pages and Web sites by mastering HTML5 coding techniques...
    798930ae-2d16-45f2-8306-734fc7f5a22b-0d7af0d752c8.small Databases: OLAP and Recursion
    The On-Line Analytical Processing section of this course introduces star schemas...
    B01ee61e-1ac1-4a07-b5f4-348a4b4868d6-934315464fc2.small Databases: Semistructured Data
    This course includes the following components: XML Data; JSON Data; XPath and...

    © 2013-2019