Visualizing Gradient Descent Methods      


Visualizing Interactive Gradient Descent

Dao Ming and Nova Zhang



Abstract

This project seeks to address the knowledge gap users in fields associated with optimization have with regards to gradient descent. Existing solutions are largely static, or lack a self-experimentation component for mastering key visual concepts. Interactive Gradient Descent describes our hands-on and customizable visualization system that aims to educate and serve as a tool through the form of a web application. It allows the user to explore the effects of adjusting hyperparameters (e.g. learning rates), initialization points and variable domains on different feature spaces on optimality and time to convergence.

Running Instructions

See a video demo of this application here:

Video Demo Each page features:
  • A top bar with different gradient descent methods (SGD, Momentum, RMSProp, Adam, AMSgrad).
  • Click each circle to toggle the method. Hovering over each method text gives a short description of it
  • After plotting, click ‘Iteration counter’ to display a bar chart comparing the no. of iterations each method requires.

To run the descent methods, click anywhere on the feature space.

The application can be navigated using the sidebar (toggled using ‘Options’). It consists of
  • Sample functions (‘Gaussian ‘Bowl’’, ‘Rastrigin’, ‘Rosenbrock’) corresponding to different feature spaces (contour plots).
  • Custom: Allows for the user to input a given 2 variable (x,y) function, specifying the corresponding domain/range.
  • Parameters: Allows user to set the learning rate for each gradient descent method
  • About: Learn about the motivation behind the project and what gradient descent is.
Read our full paper here: Visualizing Interactive Gradient Descent