Representation Learning

AM 2: Representation Learning - from a Deep Learning Perspective
Winter Semester 2016/2017
Dr. Sebastian Stober
Mon 14-16; Golm, Haus 10, Raum 0.26

Begins: Monday, October 17

“The concept of representation learning ties together all of the many forms of deep learning. Feedforward and recurrent networks, autoencoders and deep probabilistic models all learn and exploit representations. Learning the best possible representation remains an exciting avenue of research.”
Deep Learning Book, Chapter 15: Representation Learning

Over the last decade, so-called “deep learning” techniques have become very popular in various application domains such as computer vision, automatic speech recognition, natural language processing, and bioinformatics where they produce state-of-the-art results on various challenging tasks. Most recently, deep neural networks even succeeded in mastering the game of go.

A crucial success factor of deep neural networks is their ability to learn hierarchies of increasingly complex features from raw input data. Such representations often outperform traditionally hand-crafted features that require expensive human effort and expertise. But they do not come as a free lunch. Designing and training deep neural networks such that they actually learn meaningful and useful feature representations of data is an art itself. Mastering it requires practice and experience - and this course offers many opportunities to obtain these.

This is an intense course following a “flipped classroom” design that builds on everybody’s active participation. You will be responsible to prepare for each class by reading selected literature or watching online video lectures and talks. Each student will further reflect on their learning progress in a brief weekly blog post. This can, for instance, include open questions, self-discovered answers or pointers to additional helpful material. About 50% of the in-class time will be used to discuss the open questions about the concepts under study. In the remaining time, we will focus on advancing the course project by applying what you just learned and developing new ideas.

As course project, you will design, train and analyze an automatic speech recognition system. This will be a collaborative team effort with a little competition about which team eventually learns the best representation. For the project, you will have access to a professional-grade GPU compute server and a dataset of approximately 1000 hours of transcribed audio recordings. With this you will able to build a proper system. The implementation will be in Python using either Theano or Tensorflow as base framework. Each team will document their progress in weekly blog posts with rotating authors and describe their final solution in a short paper.

The course primarily addresses students within the Cognitive Systems Master but is also open to a general audience. As prerequisites for the course, you should already have basic knowledge on machine learning in general as well as linear algebra and probability theory. Those who would like a refresher prior to taking this class can find an excellent compilation in Part I of the Deep Learning Book. You should also know already how to program in Python and of course bring some curiosity to embark on an open-ended and exploratory learning process for the course project.

If you have got questions about the course, please contact Sebastian Stober. There is also a Campu.UP workspace for the course to discuss questions and help each other!