报告题目：Spectrum concentration in deep residual learning: a free probability approach

报 告 人：Prof. Qiu Caiming

报告时间：2018年12月03日 上午10:30

报告地点：主楼III-401

报告摘要：We revisit the weight initialization of deep residual networks  (ResNets) by introducing a novel analytical tool in free probability to the community of deep learning. This tool deals with the limiting spectral distribution of \textit{non-Hermitian} random matrices, rather than their conventional \textit{Hermitian} counterparts in the literature.  As a consequence, this new tool enables us to evaluate the singular value spectrum of the input-output Jacobian of a fully-connected deep ResNet in both linear and nonlinear cases. With the powerful tool of free probability, we conduct an asymptotic analysis of the (limiting) spectrum on the single-layer case, and then extend this analysis to the multi-layer case of an arbitrary number of layers. The asymptotic analysis illustrates the necessity and university of rescaling the classical random initialization by the number of residual units $L$, so that the mean squared singular value of the associated Jacobian remains of order $O(1)$, when compared with the large width and depth of the network. We empirically demonstrate that the proposed initialization scheme learns at a speed of orders of magnitudes faster than the classical ones, and thus attests a strong practical relevance of this investigation.

报告人简介：

 Prof. Qiu Caiming, IEEE Fellow, Specially-appointed Professor of National “Thousand Talents Program”, Distinguished Professor of “Thousand Talents Program” of Shanghai, Chair Professor of Shanghai Jiaotong University, Director of Big Data Engineering Research Center of Shanghai Jiaotong University, Tenured professor of the Tennessee Institute of Technology, USA.
Professor Qiu Caiming received a bachelor's degree in science from Xidian University in 1987, a master's degree from the University of Electronic Science and Technology in 1990, and a doctor's degree from the Institute of Technology, New York U