Front Page

CS [L,W]182/282A Designing, Visualizing and Understanding Deep Neural Networks Spring 2020

New YouTube Video Location:

The class will be recorded and available on youtube. Here are the new coordinates:
Lecture Playlist: https://www.youtube.com/playlist?list=PLnocShPlK-FvSQvoTWZuJQzEiDDAY64kT Links to an external site.

and new zoom lecture link Zoom 434-690-973 Links to an external site.

Important: New Course Format

This semester, 182/282A will be a new format. The "L" section has live lectures with mandatory attendance. The "W" section has only recorded or live-streamed lectures. Students in the "W" section should not attend the "L" section lectures. The lectures themselves are identical. The rest of the course material, assignments, midterms, discussion sections, Piazza etc. are identical.

iClickers

If you're in one of the in-class sections, please acquire an iClicker (any model from iCLicker+, iClicker2 etc will work) and register it (the registration link is on the left) by class on 2/4/20. If you're not in a live section, you dont need an iClicker (for grads you are not in a live section unless you requested to be).

Course Information

Lectures: 9:30am-11am TuTh in 306 Soda. Please attend lecture (including the first one) only if you are enrolled in the "L" section of 182. The "W" section and 282A are online only.

Reporting Issues

Please read this page before posting.

Piazza

Use this link (Links to an external site.) to Piazza for general class discussions

Logistics

Prerequisites

The prerequisites for this course are:

* Knowledge of calculus and linear algebra, Math 53/54 or equivalent. You'll need this throughout the course.

* Probability and Statistics, CS70 or Stat 134. We'll talk about continuous and discrete probability distributions. CS70 is bare minimum preparation, a stat course is better.

* Machine Learning, CS189. You may be able to manage the course without 189 but in that case you should have a strong stat background.

* Programming, CS61B or equivalent. Assignments will mostly use Python. If you need some help, try this tutorial from CS231n (Links to an external site.)Links to an external site.

Texts

We'll frequently use the online book: Deep Learning (Links to an external site.)Links to an external site. by Ian Goodfellow and Yoshua Bengio and Aaron Courville. For reinforcement learning, the new version of Sutton and Barto's classic book is available online (Links to an external site.)Links to an external site..

Grading

  • Class Participation: 10%
  • Midterms: 30%
  • Final Project (in groups): 30%
  • Assignments : 30%

Slip Days

You can use up to 8 slip days for late assignments.

Office hours

All Office Hours + Discussion can also be found in this calendar link Links to an external site..

Staff Time Location
John Canny M 2-3pm Zoom 434-690-973 Links to an external site.
Aravind Srinivas M 1-3pm
https://berkeley.zoom.us/j/3151083806 Links to an external site.
David Chan W 8-9:30am
F 8-9:30am
Forrest Huang Tu 4-5pm https://berkeley.zoom.us/j/616887975 Links to an external site.
Philippe Laban W 12-1pm
Roshan Rao M 9-11am https://berkeley.zoom.us/j/506306778 Links to an external site.
Haozhi Qi F 9:00-11am
Michael-David Sasson Tu  2-3P 315 Soda (for administrative questions)

Lectures Online

The class will be recorded and available on youtube. Here are the new coordinates:
Lecture Playlist: Links to an external site.https://www.youtube.com/playlist?list=PLnocShPlK-FvSQvoTWZuJQzEiDDAY64kT Links to an external site. Links to an external site.

Please use this link for live streaming: zoom lectures Links to an external site.

In addition, the powerpoint slides posted *after* class will include audio and inline questions, and will be labelled "lecXX_voice.pptx". We recommend using the slides rather than the video to watch lectures.

Finally, there are pdf files without animation and without sound, however the embedded questionaire links still work.

Schedule

 Date           
Lecture Topic                                                       
Reading                  
Assignments/Section Notes                         
Tu 1/21 Introduction, Course Overview, Brief history of deep networks. Download pptx or Download pdf Introduction Links to an external site. from Deep Learning Links to an external site.
Th 1/23 Machine learning concepts: Loss and Risk, Discriminative models, Linear and Logistic Regression. Download pptx or Download pdf Review: sections 1.1-1.3 and 6.6-6.9 from the Download CS189 book (skip KL-div). Do Python/Numpy tutorial Links to an external site. if needed. Section 1 notes Download pdf
Tu 1/28 Bias-Variance tradeoff, Regularization, SVMs, Multiclass classification, Softmax. Cross-validation. Download pptx and Download pdf sections 1.4-1.6, 6.10-6.11, from the Download CS189 book (skip Tikhonov) Assignment 1 out
Th 1/30 Optimization, Stochastic Gradient Descent. Download pptx and Download pdf Chapter 8 Links to an external site. of Deep Learning Links to an external site.
Optimization Notes Links to an external site.
Section 2 notes Download pdf
Tu 2/4 Backpropagation, Convolutional Networks. Download pptx and Download pdf
Backpropagation Notes Links to an external site.
Convnet notes Links to an external site.
Th 2/6 CNN examples, Activation functions, Initialization. Download pptx and Download pdf Convnet notes Links to an external site.
Training Neural Networks 1 Links to an external site.
Training Neural Networks 2 Links to an external site.
CS282A Project Proposal out 
Tu 2/11 Training: Batch normalization, Dropout, Ensembles, Hyperparameter tuning. Download pptx and Download pdf Training Neural Networks 2 Links to an external site.
Training Neural Networks 3 Links to an external site.
Section 3 Notes Download pdf
Th 2/13 Object Detection and Segmentation. Download pptx and Download pdf Introduction to Object Detection Links to an external site. (All sections except ParseNet, PSPNet, DeepLab, PANet and EncNet)   Introduction to Semantic Segmentation Methods Links to an external site.
Mon 2/17 CS182 Project Proposal out
Tu 2/18 Recurrent Networks, LSTMs, applications. Download pptx and Download pdf RNN chapter Links to an external site. from Deep Learning Links to an external site.
Understanding LSTM Networks Links to an external site.
CS282A Project proposal due
Download MT1 SP19 
Download MT1 SP19 Solutions
Download Practice MT1
Older Download MT1
Section 4 Notes Download pdf
Wed 2/19 Assignment 1 due 11pm
Assignment 2 out
Th 2/20 Visualizing Deep Networks. Download pptx and Download pdf Links to an external site.Understanding Neural Networks Through Deep Visualization Links to an external site.
Feature Visualization Links to an external site.
Mon 2/24

CS182 Project Proposal due

Mon 2/24 MT1 Review Session

8-10P in Pimentel

Tu 2/25 Attention Networks. Download pptx and Download pdf Visual Attention Models Links to an external site.
Paper: Recurrent Models of Visual Attention Links to an external site.
Download Practice Mid-term Solutions
W 2/26, 7-8:30P Midterm 1 Locations
Th 2/27 Semantic Models for Text. Download pptx and Download pdf Links to an external site.Blog: WordToVec Links to an external site.
Paper: WordToVec Links to an external site.
Blog: Skip-Thought Vectors Links to an external site.
Paper: Skip-Thought Vectors Links to an external site.
Tu 3/3 Natural Language Translation and Transformers Download pptx and Download pdf Links to an external site.The Illustrated Transformer Links to an external site.
NMT by Jointly Learning to Align and Translate Links to an external site.
Attention is All You Need Links to an external site.
Section 5 notes Download pdf
Th 3/5 Pretraining Language Models. Download pptx and Download pdf The Illustrated Transformer
The Illustrated GPT-2
Assignment 2 due 11pm
Assignment 3 out
Tu 3/10 Dialog and other Applications pdf, doc Links to an external site. and zoom recording Links to an external site.Blog: BERT Links to an external site.
Paper: BERT: Pre-training of Deep Bidirectional Transformers Links to an external site.
Section 6 notes Download pdf
Th 3/12 Generative Models Download pptx (corrected) , Download pptx voice and Download pdf Links to an external site.Understanding VAEs Links to an external site.
Autoregressive Networks Links to an external site.
Paper: Variational Auto-Encoders Links to an external site.
Paper: Autoregressive Models Links to an external site.
Paper: Image Transformer Links to an external site.
Project Checkpoint 1
due week of 3/30/20
Tu 3/17 Generative Adversarial Networks Download pptx Download pdf and zoom rec Links to an external site. Links to an external site.What is a GAN? Links to an external site.
Paper: Generative Adversarial Networks Links to an external site.
Paper: DCGAN Links to an external site.
Wasserstein GAN Blog Post Links to an external site. (From GAN to WGAN)

Download Practice Midterm 2

Download Practice Midterm 2 Solutions


Section 7 notes Download pdf

Th 3/19 Adversarial and Fooling Networks. Download pptx and Download pdf and lecture recording (google drive) Links to an external site. Links to an external site.Attacking Models with Adversarial Examples Links to an external site.
Breaking Neural Networks with Adversarial Attacks Links to an external site.
Paper: Adversarial Examples Links to an external site.
Paper: Transferability/blackbox attacks Links to an external site.
Paper: Physical perturbation Links to an external site.

Assignment 4 out
Tu 3/31 Fairness in Deep Networks Download pptx , Download pptx voice, Download pdf and zoom rec Fairness in ML Blog Links to an external site.
Blog: MINE Links to an external site.
Paper: MINE: Mutual Information Neural Estimation Links to an external site.
Assignment 3 due 11pm
Th 4/2

Imitation Learning pptx, pptx voice, Download pdf

and video Links to an external site.

RL - Imitation Learning Links to an external site.
Paper: End-to-End Learning for Self-Paper: Driving Cars Links to an external site.
Paper: DAGGER Links to an external site.
Paper: Adversarial Imitation Learning Links to an external site.
Mo 4/6 Take-home quiz 1 out
Tu 4/7 Reinforcement Learning: Policy Gradients pptx Download pdf pptx_voice and zoom video Links to an external site. Policy Gradient Methods, chapter 13 of "Reinforcement Learning" Links to an external site.
Th 4/9 Reinforcement Learning:
Value-based methods pptx pptx_voice, Download pdf and zoom video Links to an external site.
Beating Video Games with Deep Q Learning Links to an external site.
Paper: DQN paper (Nature) Links to an external site.
Paper: Asynchronous Methods for Deep RL Links to an external site.
Paper: Double DQN Links to an external site.
Take-home quiz 1 due
Tu 4/14 Exploration pptx pptx voice Download pdf and zoom rec Links to an external site. Exploration in Reinforcement Learning Links to an external site.
Curiosity Driven Learning Through Next State Prediction Links to an external site.
Paper: Count-Based Exploration and Paper: Intrinsic Motivation Links to an external site.
Paper: Curiosity-driven Exploration Links to an external site.
Paper: Infobot: Information Bottleneck for RL Links to an external site.
Section 8 notes Download pdf
Th 4/16 Unsupervised Learning pptx, pptx voice, and zoom rec Links to an external site.

Self-Supervised Learning Links to an external site.
Paper: Split Brain Autoencoder Links to an external site.
Paper: Jigsaw Links to an external site.
Paper: Contrastive Predictive Coding Links to an external site.
Links to an external site.Paper: Learning to Poke by Poking

Section 9 notes Download pdf
Mon 4/20 Assignment 4 due 11pm
Take-home quiz 2 out
Tu 4/21 Playing Games Download pptx Download pptx voice and Download pdf and zoom rec Links to an external site. A Simple Apha(Go) Zero Tutorial Links to an external site.
Paper: Mastering the Game of Go without Human Knowledge Links to an external site.
Th 4/23 Neural Architecture Search / Auto-ML Download pptx and zoom rec Links to an external site.

Neural Architecture Search - The AutoML Process Links to an external site.
Paper: Learning Transferable Architectures for Scalable Image Recognition (NASnet) Links to an external site.

Take-home quiz 2 due
Tu 4/28 Graph Neural Networks Download pptx , Download lec27-voice.pptx, zoom rec.mp4 Links to an external site.

Graph Neural Network Tutorial 1 Links to an external site.

Graph Neural Network Tutorial 2 Links to an external site.

Paper: Semi-supervised classification with graph convolutional networks Links to an external site.

Th 4/30 Deep Networks and Health pptxpptx with voice, zoom rec Links to an external site.

Tutorial: NeurIPS 2019 Computational Biology + Health Links to an external site.
Blog: AlphaFold Links to an external site.
Paper: Low-N Protein Engineering Links to an external site.
Paper: Cardiac Arrest Prediction Links to an external site.
Paper: RL for Ventilator Weaning Links to an external site.
Paper: Importance of Explainable Models Links to an external site.

We 5/13 Project Report due 11pm