Assignments 7: Optimization Techniques

Basic Reading

From the Deep Learning Book - Chapter 8: Optimization for Training Deep Models, please read up to (including) section 8.5 focussing on the following parts:

• 8.2 Challenges in Neural Network Optimization

• 8.3 Basic Algorithms

• 8.4 Parameter Initialization Strategies

• 8.5 Algorithms with Adaptive Learning Rates

Here are some points to guide you during reading:

• What is the rationale behind the different techniques? Which problems / challenges are addressed?

• What are typical application scenarios?

• Take note of possible advantages and disadvantage as well as potential pitfalls!

• Think about, whether and how the described techniques could be applied in the context of the course project!

Post your remarks, insights and open questions in your blog and in the forum! Help each other to make sure you are not getting stuck too long with some unresolved issue.

Deadline for questions to be considered in class is December 10, 7am. I will also try to accommodate things that come in later but I cannot make guarantees. The earlier you bring up questions, the better.

Further Reading

This week, there is no mandatory further reading. However, you are invited to explore the vast deep learning literature and post your findings in your blog and the dedicated forum threads.

Right now, many highly interesting deep representation learning papers are being published …

• in the Proceedings of the 2016 NIPS conference

• as submissions (currently under review) for the 2017 International Conference on Learning Representations (ICLR)

Course Project

Please share your code and presentations from the concept projects (previous assignments) in Campus.UP - either as links or uploaded documents!

Furthermore, each group from the Unconference should open a new thread in the course forum and post their minutes. Then proceed to take first steps as discussed - distributing workload as needed over those contributing! Work should be coordinated through the forum, so others not directly involved can follow up.

For the next meeting in class, each group should prepare a short progress report (1-3 min, not more!) with the main points - preferably as a blog/forum post, slide or jupyter-notebook. Please avoid browsing through code at this point! Additionally, you can prepare questions or points you would like to discuss in class. It helps for preparation if these are also posted in Campus.UP a little earlier.