Deep neural networks have been widely used in massive engineering domains, but the training and deployment of neural networks are subject to many fundamental challenges. In the training phase, the large-scale optimization often consumes a huge amount of computing and energy resources. In practical deployment, we often need the capability of uncertainty quantification to ensure the safe operations in an uncertain environment. To address the first challenge, we need compressed training, but it is hard to determine the compression ratio automatically in the training phase. To address the second challenge, we often use Bayesian learning models, but the resulting uncertainty-aware model often leads to massive model copies which cause huge memory and computing overhead.

In this talk, we show that the interplay of compressed training and Bayesian learning can provide more sustainable neural network models. Firstly, we investigate end-to-end tensor compressed training. This approach can offer orders-of-magnitude parameter reduction in the training phase, but it is hard to determine the tensor rank and model complexity automatically. We show that efficient Bayesian formulation and solver can be developed to address this major challenge, enabling high-accuracy end-to-end compressed training as well as energy-efficient on-device training. Secondly, we investigate MCMC-type Bayesian training. Here the main challenge is how to use a small number of model copies to accurately represent model uncertainties. We provide an online and provable online sample thinning method based on kernelized Stein discrepancy. This method can reduce the model copies on the fly, and offers orders-of-magnitude memory and latency savings in the inference.

 

Speaker’s Bio:

Dr. Zheng Zhang is an Assistant Professor of Electrical and Computer Engineering at University of California, Santa Barbara. He received his PhD degree in Electrical Engineering and Computer Science from MIT in 2015. His research is focused on uncertainty quantification and tensor computation, with applications to multi-domain design automation, sustainable and trustworthy AI systems. He received the ACM SIGDA Outstanding New Faculty Award, IEEE CEDA Early CAREER Award, NSF Early Career Award, and three best journal paper awards from IEEE Transactions in the EDA research field. He is the receipt of ACM SIGDA Outstanding Dissertation Award in 2016, and MIT Microsystems Technology Lab PhD Dissertation Award in 2015.