A series of sixteen lectures covering the core of the book "Information Theory, Inference, and Learning Algorithms (Cambridge University Press, 2003)" which can be bought at Amazon, and is available free online. A subset of these lectures used to constitute a Part III Physics course at the University of Cambridge.

Introduction to information theory

* The possibility of reliable communication over unreliable channels. The (7,4) Hamming code and repetition codes.

Entropy and data compression

* Entropy, conditional entropy, mutual information, Shannon information content. The idea of typicality and the use of typical sets for source coding. Shannon's source coding theorem. Codes for data compression. Uniquely decodeable codes and the Kraft-MacMillan inequality. Completeness of a symbol code. Prefix codes. Huffman codes. Arithmetic coding.

Communication over noisy channels

* Definition of channel capacity. Capacity of binary symmetric channel; of binary erasure channel; of Z channel. Joint typicality, random codes, and Shannon's noisy channel coding theorem. Real channels and practical error-correcting codes. Hash codes.

Statistical inference, data modelling and pattern recognition

* The likelihood function and Bayes' theorem. Clustering as an example

Approximation of probability distributions

* Laplace's method. (Approximation of probability distributions by Gaussian distributions.)

* Monte Carlo methods: Importance sampling, rejection sampling, Gibbs sampling, Metropolis method. (Slice sampling, Hybrid Monte Carlo, Overrelaxation, exact sampling)

* Variational methods and mean field theory. Ising models.

Neural networks and content-addressable memories

* The Hopfield network.

Dates:

- Free schedule

Included in selections:

Machine Learning

Machine learning: from the basics to advanced topics. Includes statistics...

Machine learning: from the basics to advanced topics. Includes statistics...

Математическое и имитационное моделирование

3 курс МИЭМ ВШЭ, 5 кредитов.

3 курс МИЭМ ВШЭ, 5 кредитов.

More on this topic:

Monte Carlo Methods in Finance

Non-bankers can learn to understand the mathematical models that have made the...

Non-bankers can learn to understand the mathematical models that have made the...

Analytics of Finance

This course covers the key quantitative methods of finance: financial econometrics...

This course covers the key quantitative methods of finance: financial econometrics...

Information Theory

This course is an introduction to information theory, which emphasizes fundamental...

This course is an introduction to information theory, which emphasizes fundamental...

Principles of Digital Communication II

This course is the second of a two-term sequence with 6.450. The focus is on...

This course is the second of a two-term sequence with 6.450. The focus is on...

Atomistic Computer Modeling of Materials (SMA 5107)

This course uses the theory and application of atomistic computer simulations...

This course uses the theory and application of atomistic computer simulations...

More from 'Mathematics, Statistics and Data Analysis':

Derivatives Markets: Advanced Modeling and Strategies

Financial derivatives are ubiquitous in global capital markets. Students will...

Financial derivatives are ubiquitous in global capital markets. Students will...

Doing Journalism with Data: First Steps, Skills and Tools

This free 5-module online introductory course gives you the essential concepts...

This free 5-module online introductory course gives you the essential concepts...

Big Data Mini Course: AMP Camp 4 hands-on exercises

The exercises we cover today will have you working directly with the Spark specific...

The exercises we cover today will have you working directly with the Spark specific...

Digital Analytics Fundamentals

This three-week course provides a foundation for marketers and analysts seeking...

This three-week course provides a foundation for marketers and analysts seeking...

Introduction to Machine Learning (CSC2515, Fall 2008)

Introductory course in machine learning by world leading expert Geoffrey Hinton...

Introductory course in machine learning by world leading expert Geoffrey Hinton...

© 2013-2019