We study the benign overfitting phenomenon induced by simple optimization algorithms in deep learning. Oftentimes the neural network is over-parameterized in the sense that the number of parameters exceeds the training data size, but the obtained solution generalizes well to unseen data. The generalization stems from an implicit regularization of the optimization algorithm. We present the recent theoretical development of over-parameterization for linear/non-linear models, together with some numerical experiments.