Data-Driven Model Reduction, Scientific Frontiers, and Applications
- Zhangyang (Atlas) Wang, Department of Computer Science & Engineering
- Embracing Sparsity in Deep Networks: From Algorithms to Hardware
In this talk, I will present how the sparsity, a desirable property for both algorithms and hardware design, could be discovered, exploited, and enhanced in the context of deep networks. I will first introduce how an iterative sparse solver could be linked to and tuned as a feed-forward deep network, using the “unfolding then truncating” trick. Next, I will show how a double sparse dictionary structure could be naturally utilized to sparsify the weights of the obtained networks resulting in a new deep network whose feature and parameter spaces are simultaneously sparsified. I will then describe PredictiveNet, a recent work by one of my collaborators, which predicts the zero hidden activations of the nonlinear CNN layers at low costs thereby skipping a large fraction of convolutions in CNNs at runtime without modifying the CNN structure or requiring additional branch networks. Such an energy-efficient hardware implementation could be seamlessly integrated with the theory bridging sparse optimization and deep learning, potentially leading to even larger energy savings.