Ongoing Performance Monitoring
MRM Performance Monitoring | MRM
Ongoing performance monitoring (OPM) is a critical component of successful model risk management (MRM). Regulators have been focusing more attention on this aspect of model lifecycle management due to notable inconsistencies observed during recent examinations. Implementing an effective OPM process that confirms a model remains fit for purpose and continues to function as intended between validation intervals provides stakeholders with continued confidence in the models relied upon for decision making and awareness of when models are in need of adjustment or replacement.
When monitoring the performance of our models, we want to answer the following two questions: 1) is the model’s output reliable (is it producing stable and consistent results), and 2) is the model’s output valid (is it accurately measuring what it is supposed to measure). The distinction is important since you can have a model with stable/consistent results but results that are not valid or representative of what we are trying to measure. For example, we can utilize a 3rd-party loan prepayment model as part of our Current Expected Credit Loss (CECL) modeling that produces stable and consistent results. However, the results may be materially off when compared to your institution’s actual prepayment experience. Note while this model’s output could be considered reliable but not valid, it could remain useful if quantitatively adjusted with an overlay to produce reasonable/valid results.
OPM in practice can be challenging as it requires continuous effort on the part of model owners or developers, the establishment of individualized testing processes and key risk indicators/triggers, execution of the testing, effective communication of results, and MRM monitoring and follow up. In addition, there is no “one-size-fits-all” approach institutions can adopt – the nature and frequency of the monitoring is a function of each model’s design, complexity, and performance along with availability of new data or modeling approaches, and the magnitude of the risk involved.
When establishing an OPM process for a given model, begin by looking at the specific testing that was performed during development or implementation and consider repeating these tests on some recurring basis. This can include elements of data testing, statistical testing, benchmarking, backtesting, and sensitivity analysis. When establishing your testing approach, be sure to identify and document the following:
The specific tests to be performed
The frequency of this testing
Who is responsible for the testing
The thresholds or triggers that, if breached, would drive additional communication and corrective action
The action steps that would be taken in the event thresholds were breached.
In addition, make sure your testing reassesses any model limitations that may have been identified during development.
When it comes to statistical models, there are a number of tests commonly performed to confirm a model’s ongoing validity and reliance. One method is to evaluate a model’s coefficient stability. This is determined by introducing new data (perhaps a new year’s worth of historical data) and evaluating the change in a variable’s coefficient and standard error – a noticeable change could indicate weakness in the model’s reliability or predictive power. As part of this testing, you can also evaluate the stability of the p-values and the significance – again, a noticeable change here could identify model weakness. Other measures commonly employed for statistical model testing include: the Gini Coefficient, Population Stability Index, Type I and Type II error metrics, K-fold tests, and Walk-forward tests.
The frequency of these tests should align with the type of model, frequency of data updates, and results from prior performance tests. For example, a mature capital/credit stress testing model that has performed well over time and is used and updated annually may only require testing on an annual basis whereas a CECL model that is new, untested, and updated with new data quarterly may require more frequent testing.
For non-statistical models, techniques can include a re-evaluation of alternative methodologies previously considered during development, backtesting or outcomes analysis (comparing actual vs. forecasted results), benchmarking (replicating and comparing the calculations with an alternative model or spreadsheet), and sensitivity testing (adjusting key variables or inputs to determine how sensitive results are to the incremental changes made). The latter method can also be particularly effective when evaluating 3rd party/“black box” models where the underlying mathematics or logic are not fully disclosed or apparent. Note: for more specific examples of OPM with non-statistical models such as asset/liability management or liquidity models, see companion articles within this MRM Insights publication.
While OPM is a core element of model validation, it is not necessarily the responsibility of MRM (your second line of defense) to perform – in most cases, the model owner is best equipped to facilitate the process given the knowledge and experience using the model, updating data and assumptions, and reviewing results. However, when skills or resource availability preclude model owners from performing this important analysis, MRM or other internal/external resources can be called upon to assist.
A model’s performance will deteriorate over time no matter how good a model is – real-world conditions change compared to historical observations and experience. The OPM process provides stakeholders with a way to periodically evaluate a model’s performance, to indicate when it is deteriorating, and to drive intervention when needed.
Learn more about our Model Risk Management services.
ABOUT THE AUTHOR
Michael R. Guglielmo is a Managing Director at Darling Consulting Group. With over 30 years of experience in strategic risk management, Mike has provided technical and strategic consulting to a diverse group of financial institutions. Mike is also a frequent author and top-rated speaker on a variety of balance sheet and model risk management and operational risk management topics.
Contact Michael Guglielmo: mguglielmo@darlingconsulting.com or 978-499-8159 To find out how DCG can help your institution optimize its OPM.
© 2021 Darling Consulting Group, Inc.
