Deep Networks have revolutionized computer vision, speech recognition and language translation. They have growing impact in many areas of science and engineering. They also do not follow a closed set of theoretical principles. In Yann Lecun's words they require "an interplay between intuitive insights, theoretical modeling, practical implementations, empirical studies, and scientific analyses." This course attempts to cover that ground, and has three goals:

* Design principles and best practices: design motifs that work well in particular domains, structure optimization and parameter optimization.

* Visualizing deep networks. Exploring the training and use of deep networks with visualization tools.

* Understanding deep networks. Methods with formal guarantees: generative and adversarial models, tensor factorization.

This is a first offering of the class, and it draws heavily from the excellent Stanford course CS231n: Convolutional Neural Networks for Visual Recognition. by Li, Karpathy and Johnson.

Projects:

The public projects from this semester are up now.

Logistics

Course Number: CS 294-129 Fall 2016, UC Berkeley
CCN: 34652
Instructor: John Canny
GSIs:
- Yang You: youyang@cs.berkeley.edu
- Daniel Seita: seita@berkeley.edu
Time: MW 1pm - 2:30pm
Location: Soda 306
Discussion: Join Piazza for announcements and to ask questions about the course
Office hours:
- John Canny - M 2:30-3:30, in 637 Soda
- Yang You - WF 9:00-10:00AM in 576 Soda
- Daniel Seita - Thurs 5:00-6:00PM in Visual Computing Lab (5th floor Soda Hall)

Lectures Online

Lectures will be live streamed at this link.

After class, they will be archived here.

Prerequisites

The prerequisites for this course are:

* Knowledge of calculus and linear algebra, Math 53/54 or equivalent. You'll need this throughout the course.

* Probability and Statistics, CS70 or Stat 134. We'll talk about continuous and discrete probability distributions. CS70 is bare minimum preparation, a stat course is better.

* Machine Learning, CS189. You can probably handle this course without a machine learning course before, but it will be very helpful.

* Programming, CS61B or equivalent. Assignments will mostly use Python. If you need some help, try this tutorial from CS231n

Texts

We'll frequently use the online book: Deep Learning by Ian Goodfellow and Yoshua Bengio and Aaron Courville

Grading

Class Participation: 10%
Midterm: 20%
Final Project (in groups): 40%
Assignments : 30%

Slip Days

You can use up to 5 slip days for late assignments.

Schedule

Date	Lecture Topic	Reading	Assignments
W 8/24	Introduction to Deep Learning and Applications, Course Overview pptx pdf	Introduction from Deep Learning
M 8/29	Brief History of Computer Vision, Classification, k-Nearest Neighbors pptx pdf	Review: chapters 1-4 of Deep Learning and do Python/Numpy tutorial if needed. Image Classification Notes
W 8/31	Linear Classification, Feature selection pptx pdf	Linear Classification Notes
M 9/5	Admin Holiday
W 9/7	Optimization, Stochastic Gradient Descent pptx pdf	Chapter 8 of Deep Learning Optimization Notes
M 9/12	Backpropagation pptx pdf	Backpropagation Notes
W 9/14	Training DNNs 1: activation functions, initialization, gradient flow, batch normalization pptx pdf	Training Neural Networks 1 Training Neural Networks 2 Training Neural Networks 3	Assignment 1 due 10pm extended to Friday 9/16
M 9/19	Convolutional Networks: Convolution/pooling layers, network design, theory - Alyosha Efros guest lecture pptx pdf	Convnet notes
W 9/21	Training DNNs 2: parameter updates, ensembles, dropout pptx pdf	Training Neural Networks 3
M 9/26	Convnets for classification and Detection. Visual challenge datasets. pptx pdf	Convnet notes	Project proposal due 10pm
W 9/28	Recurrent Networks, LSTMs, applications pptx pdf	RNN chapter from Deep Learning
F 9/30			Assignment 2 due 10pm
M 10/3	Adversarial Networks Ian Goodfellow guest lecture keynote pdf	Generative Adversarial Networks Optional: Unsupervised Representation Learning with Deep Generative Adversarial Networks Improved techniques for training GANs
W 10/5	Visualizing Deep Networks Andrej Karpathy guest lecture google doc	Quite a few visualizations will be covered. Browse this list.
M 10/10	Neural Models for Text pptx pdf	Skip-Thought Vectors Optional: RNN Translation with Attention Siamese Networks for Text
W 10/12	Attention networks pptx pdf	Recurrent Models of Visual Attention Neural Machine Translation by Jointly Learning to Align and Translate
M 10/17	Deep reinforcement learning I Sergey Levine guest lecture pdf	A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning Optional: PLATO: Policy Learning through Adaptive Trajectory Optimization End-to-End Training of Deep Visuomotor Policies
W 10/19	Deep reinforcement learning II John Schulman guest lecture pdf	Pong from Pixels The Derivative Trick Asynchronous Methods for Deep Reinforcement Learning
F 10/21			Project Checkpoint 1 due 10pm
M 10/24	Practical issues in training networks pptx pdf
W 10/26	Probabilistic models pptx pdf	Probabilistic Models chapter from Deep Learning	Assignment 3 due 10pm
M 10/31	Project presentations I		Project Checkpoint Presentation due 10pm
W 11/2	Project presentations II
M 11/7	Midterm	Sample Midterm Questions	Solutions (private)
W 11/9	Variational Autoencoders Durk Kingma guest lecture pdf	Auto-Encoding Variational Bayes Improved Variational Inference with Inverse Autoregressive Flow
M 11/14	Monte-Carlo methods pptx pdf	MCMC chapter from Deep Learning Stochastic Gradient Hamiltonian Monte-Carlo
W 11/16	Scaling up Rajat Monga guest lecture pdf	Large Scale Distributed Deep Networks TensorFlow: Large Scale Machine Learning on Heterogenous Distributed Systems
M 11/21	Neural Computers pptx pdf	Neural Turing Machines Extensions and Limitations of the Neural GPU Neural Random Access Machines	Assignment 4 due 10pm
W 11/23	Non-instruction day
Su 11/27			Final Project Presentation due 10pm
M 11/28	Final project presentations I: Note time 12:30-2:30 in 306		TensorFlow Best Practices
W 11/30	Final project presentations II: Note time 12:30-2:30 in 306
Sa 12/3			Submit Final Project Poster for printing 10pm
M 12/5	Final project poster session, 1-3pm Soda 5th floor Atrium		Final Project Poster due 10pm
W 12/14	Final project report due, 10pm		Final Project Report due 10pm
F 12/16			Course Content Survey due 10pm