Every turn of the calendar it seems that another “challenging environment” or “unprecedented event” is facing risk managers. The best tools to help mitigate these conditions are the models you use to identify and manage your institution’s risk sensitivities and exposures, but how confident are you that these models perform as intended and remain “fit for use?”
Underperforming models can result in opportunity costs as well as detrimental longer-term impacts from making the wrong decisions, which may significantly influence the ongoing success of your institution. For better, more confident decision-making, consider the three key steps of a model risk management “health check.”
Confirm the reliability of model inputs (i.e., data and assumptions) to ensure accurate model results
Ensure models remain “fit for use” under a variety of stress scenarios and a rigorous ongoing performance monitoring process
Focus on Model Risk Management (MRM) and validation to help ensure high-risk models work as intended
Step 1: Model Inputs
When performing our own personal health checks, physicians tell us that we are only as healthy as what we consume. The same can be said for models – the reliability of the output is only as good as the quality of the inputs.
It’s interesting to note that given how important data quality is to reliability, more care is often not taken to ensure data files are complete and accurate when implementing and updating models. Errors in data can not only impact account structure and segmentation, but also the cashflows driving the results of many risk models that are foundational to strategy.
Another important component of model performance is model assumptions. Outdated or unsupported assumptions can translate into unreliable outputs, and that can frequently lead to bad decision-making. Considering an IRR model, for example: there are thousands of assumptions, but only a handful that truly can impact results. Ensuring that not only are the assumptions entered into the model correctly (audit), but that they are reasonable and supportable for the institution (validation), is critical to build confidence in subsequent decisions.
Step 2: Stress Testing & Ongoing Performance Monitoring
A solid foundation of reliable data and defensible assumptions can be the building blocks for a base model whose output is a good indication of risk under current conditions. But what happens if “unprecedented and challenging” events continue in more impactful ways, or well-thought-out assumptions don’t materialize as planned?
Scenario and stress testing is important for key model assumptions, particularly deposit pricing/decay and prepayment speeds to identify the impact of customer/member behaviors under various circumstances. Testing should occur regularly, with the results included in the strategic decision-making process.
Liquidity stress testing is also a critical part of the risk management process, as it not only helps identify what may cause a liquidity crisis, but also outlines how well-prepared the institution may be to handle it (in terms of timing and resources) as well as what potential responses would be. Note that while including relief scenarios in stress testing is a valuable addition to the analysis, examiners also now expect it. This information is important to allay concerns from regulators, management, and the Board about preparedness for unexpected liquidity events. While always a critical aspect of effective risk management, the focus on stress testing has been amplified following the bank failures of early 2023.
Ongoing performance monitoring is often overlooked. Once a model is implemented, independently validated, and determined to work as intended, the tendency can be to leave it be to continue performing. But measuring and monitoring that performance is key to ensuring the model continues to provide reliable information. Whether process verification or sensitivity analyses to evaluate inputs or benchmarking or back-testing to confirm outputs, performance monitoring helps ensure models remain “fit for use” in any environment.
Step 3: Model Risk Management and Validation
So how does an organization go about making the above processes part of an institution’s risk management standard?
Model Risk Management provides structure for an organization to oversee model implementation and ongoing development and helps ensure that models are working appropriately and as intended. While MRM will strengthen internal model performance oversight, it is equally as important to ensure that model validation rigor is commensurate with the institution’s model risk ratings and provides effective challenge (defined by MRM Guidance as "Critical analysis by objective, informed parties that can identify model limitations and appropriate changes.”).
The term “model validation” is often broadly defined, and an institution should understand its goal when engaging a third-party validator. If the goal was regulatory compliance, an “audit” may have been sufficient. However, the stakes are now higher due to the increased complexity of models and the reliance on model output for strategic decision-making and risk management.
Whether seeking a first-time validation or monitoring ongoing model performance, carefully consider partnering with a validator who not only has the technical expertise and horizontal experience to ensure models are functioning at their peak, but who also has the necessary cachet to effect change in the event they are not.
Health checks for risk management are not something to be done once only. Find out where you have areas to improve and work on strengthening them throughout the year, which will make your overall process stronger. Better models lead to better decisions and better decisions lead to better bottom line performance. But that takes time and dedication to a process. Start by fixing some of the little things that can ultimately lead to big improvements.
DCG’s three-part MRM “Health Check” webinar series explored the keys to building effective models, including ensuring model inputs are reasonable and accurate, measuring model performance under a variety of potential scenarios, and effectively challenging models with validation. Click here to watch the replays or get in touch with the DCG team for more information.
For more Model Risk Management insights, click here.
ABOUT THE AUTHOR
Mark Haberland is a Managing Director at Darling Consulting Group. Mark has over 25 years providing balance sheet and model risk management education and consulting to the community and mid-size banking space. A frequent author and top-rated speaker on a wide array of risk management topics, Mark facilitates educational programs and workshops for numerous financial institutions, industry and state trade associations, and regulatory agencies.
Contact Mark Haberland: mhaberland@darlingconsulting.com or 508-237-2473 to learn more about effective challenge and Model Risk Management.
© 2024 Darling Consulting Group, Inc.
Comments