The mathematical core of deep learning is function approximation by neural networks trained on data using stochastic gradient descent. I will explain an emerging geometric framework for the analysis of this process. This includes a collection of rigorous results on training dynamics for the deep linear network (DLN) as well as general principles for arbitrary neural networks. The mathematics ranges over a surprisingly broad range, including geometric invariant theory, random matrix theory, and minimal surfaces. However, little background in these areas will be assumed and the talk will be accessible to a broad audience. The talk is based on joint work with several co-authors: Yotam Alexander, Nadav Cohen (Tel Aviv), Kathryn Lindsey (Boston College), Alan Chen, Zsolt Veraszto and Tianmin Yu (Brown).