ELEC 631 - Advanced Digital Signal Processing: Deep Learning - Spring 2015

This course will explore deep learning, multistage machine learning methods that learn representations of complex data. Over the past several years, thanks for the development of new training rules, massive computing capabilities, and enormous training data sets, deep learning systems have redefined the state-of-the-art in object identification, face recognition, and speech recognition. Examples of modern tools include: Facebook's Deep Face and Google Deep Mind.

Topics to be discussed include: Deep learning architectures, training deep learning systems, convolutional neural networks (CNN's), and applications.


Richard Baraniuk
Duncan Hall 2028, 713-348-5132, richb@rice.edu
Office hours: Tuesday 2:30-3:30 p.m.
Xaq Pitkow
Duncan Hall 2046, xaq@rice.edu
Office hours: by appointment
Ankit Patel
Duncan Hall 2050, 617-233-1937, abp4@rice.edu

Teaching Assistant:

Tan Nguyen
832-614-0285, mn15@rice.edu

Location: George R Brown W211

Time: Friday, 1 - 3:30 PM

Prerequisites: ELEC 531, ELEC 533

The course is open to graduate students from any department with some background in statistics or machine learning.

Credits: 3

Course Goals and Objectives
This is a "reading course", meaning that students will select, read and present classic and recent papers from technical literature to the rest of the class in a lively debate format. Discussions aim at identifying common themes and important trends in the field. Students will also get hands on experience with deep learning software and complete a major group project.

Course Outcomes
Deep understanding of deep learning in machine learning and its relationship to computational neuroscience.

Class Participation
Students are expected to attend, present papers, prepare a summary each week and complete a major group project.


  • Class participation (15%)
  • Paper presentations (30%)
  • Paper summaries (prepared each week, 15%)
  • Group project (40%)

Schedule (Tentative)

  • 16 January 2015: Orientation
  • 23 January 2015: Student Paper Presentations Begin
  • Early May 2015: Group Project Presentations

Paper Assignment

  • 23 January 2015: "A Logical Calculus of the Ideas Immanent in Nervous Activity" + "The Organization of Behavior" (Introduction + Chapter 4)
  • 30 January 2015: "Principles of neurodynamics; perceptrons and the theory of brain mechanisms" (Chapter 3, 4, 5)
  • 06 February 2015: "Learning representation by back-propagating errors" + "Efficient Backprop"
  • 13 February 2015: "Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position" + "Backprop to a neocognitron to do handwritten digit classification" + "Gradient-based learning applied to document recognition"
  • 20 February 2015: "Neural Networks and Physical Systems with Emergent Collective Computational Abilities" + "Long Short-Term Memory" + "Speech Recognition with Deep Recurrent Neural Networks"
  • 27 February 2015: "Information Processing in Dynamical Systems: Foundations of Harmony Theory" + "A Fast Learning Algorithm for Deep Belief Nets"
  • 13 March 2015: "Learning Deep Architecture for AI" (section 4.6, 7.1, 7.2) + "Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion" + "Reducing the Dimensionality of Data with Neural Networks"
  • 20 March 2015: "ImageNet Classification with Deep Convolutional Neural Networks" + "Maxout Networks" + "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift"
  • 27 March 2015: No class
  • 03 April 2015: No class
  • 10 April 2015: "A Probabilistic Theory of Deep Learning". This is the paper that Ankit and I have been working on for the last couple months. You can find it on arXiv at http://arxiv.org/abs/1504.00641
  • 17 April 2015: To be announced
  • 24 April 2015: To be announced

Discussion Forum
Please follow the link below to register for ELEC 631 on Piazza. We will use Piazza as our discussion forum in this course.


Rice University, MS-380 - 6100 Main St - Houston, TX 77005 - USA - webmaster-dsp@ece.rice.edu