A series of sixteen lectures covering the core of the book "Information Theory, Inference, and Learning Algorithms (Cambridge University Press, 2003)" which can be bought at Amazon, and is available free online. A subset of these lectures used to constitute a Part III Physics course at the University of Cambridge.

Introduction to information theory

* The possibility of reliable communication over unreliable channels. The (7,4) Hamming code and repetition codes.

Entropy and data compression

* Entropy, conditional entropy, mutual information, Shannon information content. The idea of typicality and the use of typical sets for source coding. Shannon's source coding theorem. Codes for data compression. Uniquely decodeable codes and the Kraft-MacMillan inequality. Completeness of a symbol code. Prefix codes. Huffman codes. Arithmetic coding.

Communication over noisy channels

* Definition of channel capacity. Capacity of binary symmetric channel; of binary erasure channel; of Z channel. Joint typicality, random codes, and Shannon's noisy channel coding theorem. Real channels and practical error-correcting codes. Hash codes.

Statistical inference, data modelling and pattern recognition

* The likelihood function and Bayes' theorem. Clustering as an example

Approximation of probability distributions

* Laplace's method. (Approximation of probability distributions by Gaussian distributions.)

* Monte Carlo methods: Importance sampling, rejection sampling, Gibbs sampling, Metropolis method. (Slice sampling, Hybrid Monte Carlo, Overrelaxation, exact sampling)

* Variational methods and mean field theory. Ising models.

Neural networks and content-addressable memories

* The Hopfield network.

Dates:

- Free schedule

Included in selections:

Machine Learning

Machine learning: from the basics to advanced topics. Includes statistics...

Machine learning: from the basics to advanced topics. Includes statistics...

Математическое и имитационное моделирование

3 курс МИЭМ ВШЭ, 5 кредитов.

3 курс МИЭМ ВШЭ, 5 кредитов.

More on this topic:

Signals, Systems and Information for Media Technology

This class teaches the fundamentals of signals and information theory with emphasis...

This class teaches the fundamentals of signals and information theory with emphasis...

Monte Carlo Methods for Inference and Data Analysis

Monte Carlo methods are a diverse class of algorithms that rely on...

Monte Carlo methods are a diverse class of algorithms that rely on...

Essential Coding Theory

This course introduces the theory of error-correcting codes to computer scientists...

This course introduces the theory of error-correcting codes to computer scientists...

Atomistic Computer Modeling of Materials (SMA 5107)

This course uses the theory and application of atomistic computer simulations...

This course uses the theory and application of atomistic computer simulations...

Fundamentals of Electrical Engineering

This course probes fundamental ideas in electrical engineering, seeking to understand...

This course probes fundamental ideas in electrical engineering, seeking to understand...

More from 'Mathematics, Statistics and Data Analysis':

Engineering Calculus and Differential Equations

Learn fundamental concepts of single-variable calculus and ordinary differential...

Learn fundamental concepts of single-variable calculus and ordinary differential...

Microsoft Professional Capstone : Big Data

Validate the skills you learned in the Microsoft Professional Program for Big...

Validate the skills you learned in the Microsoft Professional Program for Big...

Microsoft Professional Capstone : Data Science

Solve a real-world data science problem in this capstone project for the Microsoft...

Solve a real-world data science problem in this capstone project for the Microsoft...

Microsoft Professional Capstone : Artificial Intelligence

Solve a real-world artificial intelligence problem in this capstone project...

Solve a real-world artificial intelligence problem in this capstone project...

Microsoft Professional Capstone: Data Analysis

Showcase the knowledge you acquired in the Data Analysis MPP in this Capstone...

Showcase the knowledge you acquired in the Data Analysis MPP in this Capstone...

© 2013-2019