Machine learning achieves tremendous success in image classification, speech recognition, and medical diagnosis. In the meantime, machine learning also brings some interesting mathematical questions: How to solve the resulting optimization more efficiently and robustly? How to apply the machine learning technique to tackle challenging problems in mathematics. In this talk, I will view the neural networks model from a nonlinear scientific computing point of view and present some recent work on developing a homotopy training algorithm to train neural networks ``layer-by-layer'' and ``node-by-node.'' I will also review the neural network from numerical algebraic geometry point view and provide a novel initialization for ReLU networks.