Analytic Expressions for Probabilistic Moments of PL-DNN with Gaussian Input

Analytic Expressions for Probabilistic Moments of PL-DNN with Gaussian Input

Adel Bibi*, Modar Alfadly*, Bernard  Ghanem,
Analytic Expressions for Probabilistic Moments of PL-DNN with Gaussian Input",
Conference on Computer Vision and Pattern Recognition (CVPR 2018) [Oral]
Adel Bibi*, Modar Alfadly*, Bernard Ghanem
deep neural network, probabilistic moments
2018
The outstanding performance of deep neural networks (DNNs), for the visual recognition task in particular, has been demonstrated on several large-scale benchmarks. This performance has immensely strengthened the line of research that aims to understand and analyze the driving rea- sons behind the effectiveness of these networks. One impor- tant aspect of this analysis has recently gained much atten- tion, namely the reaction of a DNN to noisy input. This has spawned research on developing adversarial input attacks as well as training strategies that make DNNs more robust against these attacks. To this end, we derive in this paper exact analytic expressions for the first and second mo- ments (mean and variance) of a small piecewise linear (PL) network (Affine, ReLU, Affine) subject to general Gaussian input. We experimentally show that these expressions are tight under simple linearizations of deeper PL-DNNs, es- pecially popular architectures in the literature (e.g. LeNet and AlexNet). Extensive experiments on image classifica- tion show that these expressions can be used to study the behaviour of the output mean of the logits for each class, the interclass confusion and the pixel-level spatial noise sensi- tivity of the network. Moreover, we show how these expres- sions can be used to systematically construct targeted and non-targeted adversarial attacks.