The challenge of classification confidence estimation in dynamically-adaptive neural networks

Abstract. An emerging trend to improve the power efficiency of neu- ral network computations consists of dynamically adapting the network architecture or parameters to different inputs. In particular, many such dynamic network models are able to output ’easy’ samples at early exits if a certain confidence-based criterion is satisfied. Traditional methods to estimate inference confidence of a monitored neural network, or of inter- mediate predictions thereof, include the maximum element of the Soft- Max output (score), or the difference between the largest and the second largest score values (score margin). Such methods only rely on a small and position-agnostic subset of the available information at the output of the monitored neural network classifier. For the first time, this pa- per reports on the lessons learned while trying to extrapolate confidence information from the whole distribution of the classifier outputs rather than from the top scores only. Our experimental campaign indicates that capturing specific patterns associated with misclassifications is nontrivial due to counterintuitive empirical evidence. Rather than disqualifying the approach, this paper calls for further fine-tuning to unfold its potential, and is a first step toward a systematic assessment of confidence-based criteria for dynamically-adaptive neural network computations.