Piazza

Lectures: MW, 9-10:30 in 320 Soda

CCN: 67236

Instructor: John Canny, 637 Soda, Office Hours M 2-3pm, 637 Soda

Resources

Course Information

Topics and Readings are in the Discussions Page

Course Outline (WIP)

Date	Lecture Topic	Readings	Presenters	Assignments
8/23/17	Overview Slides (pptx) (pdf)		John Canny
8/28/17	Naturalistic Voice Interfaces Slides (pptx) (pdf)	Speech Acts Routledge Encyclopedia of Philosophy entry On the Dynamic Relations Between Common Ground and Presupposition Istvan Kecskes and Fenghui Zhang	John Canny
8/30/17	Conceptual Models, Affordances, Explanations Slides (pptx) (pdf)	The Design of Everyday Things CH1: The Psychopathology of Everyday Things, Don Norman Organising User Interfaces Around Reflective Accounts Dourish, Adler and Cantwell-Smith, Proceedings of Reflection '96 Explanation and abductive inference. Lombrozo, T., in K.J. Holyoak and R.G. Morrison (Eds.), Oxford Handbook of Thinking and Reasoning Oxford U. Press (2012). Reasoning About Object Affordances in a Knowledge Base Representation Yuke Zhu and Alireza Fathi, and Li Fei-Fei, ECCV 2014, see also Do et al., AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection, arXiv 2017	John Canny
9/6/17	Visual Description David's slides Mick's slides	Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. Xu et al. A Hierarchical Approach for Generating Descriptive Image Paragraphs. Krause et al.	David Gaddy Mick Jermsurawong
9/11/17	Video Description David's slides Allen's slides	Improving LSTM-based Video Description with Linguistic Knowledge Mined from Text, Venugopalan et al. Visual Relationship Detection with Language Priors, Lu et al.	David Chan Allen Tang
9/13/17	Visual Q&A David's Slides Mick's Slides	Andreas, Jacob, et al. "Learning to compose neural networks for question answering." arXiv:1601.01705 (2016). Xu et al. "Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering"., arXiv 1511.05234v2	David Chan Mick Jermsurawong	Project Proposal out
9/18/17	Visual Prediction and Simulation Bala's slides Vivek's Slides	Finn et al., Unsupervised Learning for Physical Interaction through Video Prediction., arXiv 1605.07157v4 Zhu et al., Visual Semantic Planning using Deep Successor Representations., ICCV 2017.	Vivek Raghuram Bala Kumaravel
9/20/17	Visual Semantic Representations Allen's slides John's slides	Luc et al. "Predicting Deeper into the Future of Semantic Segmentation", ICCV 2017. Sermanet et al. "Time-Contrastive Networks: Self-Supervised Learning from Multi-View Observation"., arXiv 2017.	Allen Tang John Canny	Project Proposal due 10pm Proposal Presentation out
9/25/17	Project Proposals	Presentation schedule		Proposal Presentation due 9am
9/27/17	Textual Q&A Meghan's slides Dequan's slides	Yih et al, Semantic Parsing via Staged Query Graph Generation: Question Answering with Knowledge Base., ACL 2015 Chen et al., Reading Wikipedia to Answer Open-Domain Questions., arXiv 2017.	Meghan Clark Dequan Wang
10/2/17	Chatbots Katie's slides Jonathan's slides	Li et al. LEARNING THROUGH DIALOGUE INTERACTIONS BY ASKING QUESTIONS, arXiv 1612.04936 Lazaridou et al. MULTI-AGENT COOPERATION AND THE EMERGENCE OF (NATURAL) LANGUAGE, arXiv 1612.07182. Please see also Vinyals et al. "A Neural Conversational Model", arXiv 1506.05869	Katie Stasaski Jonathan Cai
10/4/17	Task-Completion Agents David's slides Forrest's slides	Bordes et al. LEARNING END-TO-END GOAL-ORIENTED DIALOG., arXiv 1605.07683 Wen et al., Network-based End-to-End Trainable Task-oriented Dialogue System, arXiv 1604.04562	David Gaddy Forrest Huang
10/9/17	Memory Networks David's slides Arissa's slides	Sukhbaatar, End-To-End Memory Networks, arXiv 1503.08895 Xiong et al., Dynamic Memory Networks for Visual and Textual Question Answering., arXiv 1603.01417, see also Vinyals et al, "Pointer networks." Advances in Neural Information Processing Systems. 2015. (Link)	David Chan Arissa Wongpanich
10/11/17	Grounded Language and Action Meghan's slides An's slides	Hermann et al., "Grounded Language Learning in a Simulated 3D World." arXiv 1706.06551v2 Ghazvininejad et al., Knowledge-Grounded Neural Conversation Model., arXiv 1702.01932	Meghan Clark An Ju
10/16/17	Immitation Learning Jonathan's slides Jinkyu's slides	Yan Duan et al. One-Shot Imitation Learning., arXiv 1703.07326 Finn et al, One-Shot Visual Imitation Learning via Meta-Learning, arXiv 1709.04905 Blog post on Meta-Learning	Jonathan Cai Jinkyu Kim
10/18/17	Inverse Reinforcement Learning Katie's slides Forrest's slides	Huang et al., Enabling Robots to Communicate their Objectives, arXiv 1702.03465 Daniele et al., Navigational Instruction Generation as Inverse Reinforcement Learning with Neural Machine Translation, arXiv 1610.03164	Katie Stasaski Forrest Huang
10/23/17	Hierarchical Policies Vivek's slides David's slides	Kulkarni et al., Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation, arXiv 1604.06057 Vezhnevets et al., FeUdal Networks for Hierarchical Reinforcement Learning, arXiv 1703.01161	Vivek Raghuram David Chan
10/25/17	Robot-to-Robot Instructions David's slides Megan's slides	Rusu et al., Policy Distillation., arXiv 1511.06295 P´erez-D’Arpino et al, C-LEARN: Learning Geometric Constraints from Demonstrations for Multi-Step Manipulation in Shared Autonomy., ICRA 2017.	David Gaddy Meghan Clark
10/30/17	Grounded Spatial Maps Katie's slides David's slides	Tellex et al, Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation AAAI 2011. Paul et al., Temporal Grounding Graphs for Language Understanding with Accrued Visual-Linguistic Context., IJCAI 2017 See Also: Hu et al., Learning to Reason: End-to-End Module Networks for Visual Question Answering, ICCV 2017	Katie Stasaski David Chan
11/1/17	Semantic Maps Forrest's slides Jason's slides	Walter et al., A Framework for Learning Semantic Maps from Grounded Natural Language Descriptions, IJRR, 2014. Gupta et al., Cognitive Mapping and Planning for Visual Navigation, RSS Workshop on Spatial-Semantic Representations in Robotics, 2017.	Forrest Huang Jason Xie
11/6/17	Commanding Robots Arissa's slides Jinkyu's slides	Brooks et al., Make it so: Continuous, Flexible Natural Language Interaction with an Autonomous Robot, AAAI 2012. Karamcheti et al., A Tale of Two DRAGGNs: A Hybrid Approach for Interpreting Action-Oriented and Goal-Oriented Instructions, arXiv 1707.08668. See also Arumugam et al., Accurately and Efficiently Interpreting Human-Robot Instructions of Varying Granularities, arXiv 1704.06616	Arissa Wongpanich Jinkyu Kim
11/8/17	Asking for Help Mick's slides Forrest's slides	Deits et al., Clarifying Commands with Information-Theoretic Human-Robot Dialog, Journal of Human-Robot Interaction. Vol 2, No 2, 2013. Knepper et al., Recovering from failure by asking for help, Autonomous Robot (2015) 39	Mick Jermsurawong Forrest Huang	Last week lecture topic suggestions
11/13/17	David's slides Allen's slides	Weber et al., Imagination-Augmented Agents for Deep Reinforcement Learning https://arxiv.org/abs/1707.06203 Pascanu et al. Learning model-based planning from scratch https://arxiv.org/abs/1707.06170 Mordatch et al. "Emergence of Grounded Compositional Language in Multi-Agent Populations." arXiv:1703.04908 (2017).	David Chan Allen Tang Meghan Clark
11/15/17		Liu et al., Iterative Machine Teaching, arXiv 1705.10470 Hassibis et al., Neuroscience-Inspired Artificial Intelligence. Cell Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. arxiv 1706.02275	Vivek Raghuram Forrest Huang An Ju
11/20/17	Review slides		John Canny
11/27/17	Project Presentations	Monday Schedule		Project Presentation
11/29/17	Project Presentations	Wednesday Schedule		Project Presentation
12/1/17				Submit Project Presentation
12/06/17	Project Report			Final Project Report

Talking to Robots