The validation of machine-learning models for the stress testing of credit risk

Abstract

Banking supervisors need to know the amount of capital resources required by an institution to support the risks taken. Traditional approaches, such as regulatory capital ratios, have proven inadequate, giving rise to stress-testing as a primary tool. The macroeconomic variables that supervisors provide to institutions for exercises such as the Comprehensive Capital Analysis and Review (CCAR) programme represent a critical input into this. A common approach to segment-level modelling is statistical regression, like vector autoregression (VAR), to exploit the dependency structure between macroeconomic drivers and modelling segments. However, linear models such as VAR are unable to model distributions that deviate from normality. This paper proposes a challenger approach in the machine-learning class of models, widely used in the academic literature, but not commonly employed in practice: the multivariate adaptive regression splines (MARS) model. The study empirically tests these models using Fed Y-9 filings and macroeconomic data, released by the regulators for CCAR purposes. The champion MARS model is validated through a rigorous comparison against the VAR model, and is found to exhibit greater accuracy and superior out-of-sample performance, according to various metrics across all modelling segments. The MARS model also produces more reasonable forecasts according to quality and conservatism.