How Lightning Identifies Exploding and Vanishing Gradients
Lightning is a lightweight PyTorch wrapper for high-performance AI research that reduces the boilerplate without limiting flexibility. In this series, we are covering all the tricks Lightning offers to supercharge your machine learning training.
Training very deep networks can make your derivatives get very small or very large quickly. This problem is referred to as vanishing or exploding gradients, which makes training unstable. In this video we introduce two flags, track_grad_norm to identify vanishing and exploding gradients, and gradient_clip_val, which will clip the gradient norm computed over all model parameters together.
Follow along with this notebook: https://bit.ly/33YzC1P