Information Theory, Pattern Recognition, and Neural Networks

David MacKay, University of Cambridge

A series of sixteen lectures covering the core of the book "Information Theory, Inference, and Learning Algorithms (Cambridge University Press, 2003)" which can be bought at Amazon, and is available free online. A subset of these lectures used to constitute a Part III Physics course at the University of Cambridge.

Introduction to information theory
* The possibility of reliable communication over unreliable channels. The (7,4) Hamming code and repetition codes.

Entropy and data compression
* Entropy, conditional entropy, mutual information, Shannon information content. The idea of typicality and the use of typical sets for source coding. Shannon's source coding theorem. Codes for data compression. Uniquely decodeable codes and the Kraft-MacMillan inequality. Completeness of a symbol code. Prefix codes. Huffman codes. Arithmetic coding.

Communication over noisy channels
* Definition of channel capacity. Capacity of binary symmetric channel; of binary erasure channel; of Z channel. Joint typicality, random codes, and Shannon's noisy channel coding theorem. Real channels and practical error-correcting codes. Hash codes.

Statistical inference, data modelling and pattern recognition
* The likelihood function and Bayes' theorem. Clustering as an example

Approximation of probability distributions
* Laplace's method. (Approximation of probability distributions by Gaussian distributions.)
* Monte Carlo methods: Importance sampling, rejection sampling, Gibbs sampling, Metropolis method. (Slice sampling, Hybrid Monte Carlo, Overrelaxation, exact sampling)
* Variational methods and mean field theory. Ising models.

Neural networks and content-addressable memories
* The Hopfield network.

  • Free schedule
Course properties:
  • Free:
  • Paid:
  • Certificate:
  • MOOC:
  • Video:
  • Audio:
  • Email-course:
  • Language: English Gb


No reviews yet. Want to be the first?

Register to leave a review

Included in selections:
Small-icon.hover Machine Learning
Machine learning: from the basics to advanced topics. Includes statistics...
More on this topic:
6-895f04 Essential Coding Theory
This course introduces the theory of error-correcting codes to computer scientists...
Small-icon.hover Fundamentals of Electrical Engineering
This course probes fundamental ideas in electrical engineering, seeking to understand...
6-451s05 Principles of Digital Communication II
This course is the second of a two-term sequence with 6.450. The focus is on...
Mas-160f07 Signals, Systems and Information for Media Technology
This class teaches the fundamentals of signals and information theory with emphasis...
Mooc_statistical_mechanics_2609 Statistical Mechanics: Algorithms and Computations
In this course you will learn a whole lot of modern physics (classical and quantum...
More from 'Computer Science':
D4c7cc7d-aa4a-4e74-849c-d17417e0a649-fa9c75a8bc1f.small Introduction to Management Information Systems (MIS): A Survival Guide
Gain the skills and knowledge needed to succeed in an MIS-dominated corporate...
20e2ce7f-0e98-43eb-a956-ac820d822afd-891b8dccf7dd.small Introduction to Computational Thinking and Data Science
6.00.2x is an introduction to using computation to understand real-world phenomena...
2aa9e070-fc03-49b4-9240-b5df5d79cb15-0838bc13b7cd.small Information Security - Authentication and Access Control
Learn more fundamentals of information security, including Introduction to Cryptography...
19bbc9bb-5e0d-4205-ad7c-752a7bd87eac-b9d1304bbc7d.small The Software Architect Code: Building the Digital World
An introductory course aimed at helping you to discover and develop your software...
A0af5b37-e5fb-442f-9b95-910e57674085-a80d8eef9cf6.small Introduction to Java Programming: Writing Good Code
Learn to program in an easy and interactive way and enter the fascinating world...

© 2013-2019