Assessment of model risk in the aggregate: Contribution of quantification

Abstract

The ability to assess model risk in the aggregate is desirable, as it provides senior management with information on the overall risk associated with models used by the organisation. More importantly, it enables effective challenges and hence a robust measurement by connecting isolated information in a transparent way. This paper introduces a quantitative element to the assessment framework that utilises various model statistics from model development and performance monitoring periods. The paper links the model life cycle concept to risk quantification within families of similar models by introducing two risk measures: the Model Robustness Index (MRI) and the Model Stability Index (MSI). MRI is used to evaluate individual models’ robustness and goodness of fit. MSI employs key ongoing monitoring metrics and can be used along with judgmental factors as a dynamic measure of risk as model performance changes over time. Next, the paper introduces an approach to establish thresholds for ongoing performance monitoring. The paper demonstrates the value of these concepts by applying them to an example using binary logistic regression, that is, a simple application of a machine learning algorithm.