Ongoing Performance Monitoring

MRM Performance Monitoring | MRM

Ongoing performance monitoring (OPM) is a critical component of successful model risk management (MRM). Regulators have been focusing more attention on this aspect of model lifecycle management due to notable inconsistencies observed during recent examinations. Implementing an effective OPM process that confirms a model remains fit for purpose and continues to function as intended between validation intervals provides stakeholders with continued confidence in the models relied upon for decision making and awareness of when models are in need of adjustment or replacement.

When monitoring the performance of our models, we want to answer the following two questions: 1) is the model’s output reliable (is it producing stable and consistent results), and 2) is the model’s output valid (is it accurately measuring what it is supposed to measure). The distinction is important since you can have a model with stable/consistent results but results that are not valid or representative of what we are trying to measure. For example, we can utilize a 3rd-party loan prepayment model as part of our Current Expected Credit Loss (CECL) modeling that produces stable and consistent results. However, the results may be materially off when compared to your institution’s actual prepayment experience. Note while this model’s output could be considered reliable but not valid, it could remain useful if quantitatively adjusted with an overlay to produce reasonable/valid results.

OPM in practice can be challenging as it requires continuous effort on the part of model owners or developers, the establishment of individualized testing processes and key risk indicators/triggers, execution of the testing, effective communication of results, and MRM monitoring and follow up. In addition, there is no “one-size-fits-all” approach institutions can adopt – the nature and frequency of the monitoring is a function of each model’s design, complexity, and performance along with availability of new data or modeling approaches, and the magnitude of the risk involved.