Course Syllabus

Course Objectives

This topics course aims to present the mathematical, statistical and computational challenges of building stable representations for high-dimensional data, such as images, text and data. We will delve into selected topics of Deep Learning, discussing recent models from both supervised and unsupervised learning. Special emphasis will be on convolutional architectures, invariance learning, unsupervised learning and non-convex optimization.

Course Outline (tentative)

1st part: Convolutional Neural Networks

• Invariance, stability.
• Variability models (deformation model, stochastic model). 
• Scattering networks
• Group Formalism 
• Supervised Learning: classification. 
• Properties of CNN representations: invertibility, stability, invariance. 
• covariance/invariance: capsules and related models.
• Connections with other models: dictionary learning, LISTA.
• Other tasks: localization, regression. 
• Embeddings (DrLim), inverse problems 
• Extensions to non-euclidean domains
• Dynamical systems: RNNs.
• Guest Lecture

2nd part: Deep Unsupervised Learning

• Autoencoders (standard, denoising, contractive, etc etc)
• Variational Autoencoders
• Adversarial Generative Networks
• Maximum Entropy Distributions
• Guest Lecture

3rd part: Miscellaneous Topics

• Non-convex optimization for deep networks 
• Stochastic Optimization
• Attention and Memory Models 
• Open Problems

References

Our main resource will be a github course project 

https://github.com/joanbruna/stat212b

which will contain updated references, pointers to papers and lecture slides.

Prerequisites

Linear Algebra, Analysis, Probability, some notions of Signal Processing, and Numerical Optimization. 

Grading

This course grading will have two components:

• Paper reviewing (30%): you will be assigned two papers each, and you will be asked to produce a review following the standards of journal/conference publications. 
• Final Project (70%): It can consist in either of these three options:
• Oral presentation of a recent paper to the class,
• Tiny research project, 
• Contribute to an open-source software package (Torch, Caffe or Theano).

Final Project Proposals are due by email (joan.bruna@berkeley.edu) on April, 1st. Final projects are individual, unless there is a compelling reason for teaming up.

Office Hours

Tuesdays from 4pm to 6pm, Evans 419, or by appointment. 

Academic Honesty

Students are expected and encouraged to collaborate and share coursework. However, assignments and final projects should be conducted individually, unless there is a compelling reason to collaborate (that I should approve previously).