Good model stewardship demands policies, sound data, education, documentation, and follow-through
Risk management is part of the daily routine of a banker—after all, banks are in the business of taking risk. As part of banks' financial and operational risk assessment and decision-making processes, they need timely and relevant information to inform the evaluation of potential risks and their relative impact.
To facilitate this process, practitioners use a combination of data, models, and business judgment. Combined, these help paint an overall picture and inform everyday decisions.
Bad data or bad models lead to poor decision-making. But how do we know the data or models that we rely upon are accurate and reliable? And who bears this responsibility?
The answer: We all do!
1) Model Risk Management and the Three Lines of Defense
From the 2011 regulatory guidance on model risk management emerged the practice of the "three lines of defense:"
1. Model developers, owners, and managers
2. The model risk management function
3. Audit, along with the board of directors and examiners
Each area plays an important, collaborative role in ensuring data and models are appropriate, accurate, and well managed. Regardless of an institution's size and complexity, "effective challenge" must come from all three stakeholders – a concept that larger banks have addressed with a significant increase in resources, education, and effort over the past few years.
Community banks have been slower to adopt more substantive model risk management processes. This may be due in part to a lack of perceived risk, limited regulatory emphasis, or financial constraints.
However, as the need for—and dependence on—risk modeling and analysis grows, the potential impact of being wrong grows more profound. The responsibility of effective challenge needs to be addressed and shared across the organization.
2) Policy Development and Education
Increasingly, community bankers are developing more formalized model risk management policies that lay out an institution's MRM framework and specify the roles and responsibilities of the board to the model developers, owners, and operators. These policies typically:
Identify all the models being used by the organization (commonly called a "model inventory").
Assign a risk rating to each model (e.g., low, medium, high).
Set the established review frequency for each model by risk level and the validation methods or standards to be applied.
Establish the institution's overall model lifecycle management process. This includes development or acquisition, implementation and testing, documentation, and ongoing monitoring and retirement.
Developing the policy itself may not seem like a significant undertaking. However, what is challenging is increasing officers' and directors' awareness and understanding of model risk and their respective roles, along with implementing the various elements with model developers, owners, and managers. Getting this done requires leadership, education, time, and financial commitment.
3) Good Model Stewardship Proves Critical
The foundation of successful model risk management ultimately lies with model developers, owners, and operators. As the term "first line of defense" implies, they are on the front line of this process. They create a model or select a vendor model based upon a specified need, implement and test it, document it, and manage its ongoing use.
These steps are common and have been standard practice well before the rise of modern model risk management practices. However, organizations—particularly community banks—may struggle with:
The degree to which these activities now need to be conducted.
The level of governance and oversight associated with these actions, including data management, change control, and formal review and sign-off procedures.
The additional expectations for ongoing performance monitoring (e.g., backtesting, outcomes analysis, and variable and assumption sensitivity testing).
The level and depth of documentation.
All that said, the first line of defense (model developers, owners, and managers) holds the key to model risk management success. A well-documented, well-managed model with satisfactory performance demonstrated by meeting its intended purpose should result in a favorable model risk management assessment. This, in turn, will generate confidence among auditors, the board, and examiners.
By establishing a model risk management policy, organizations can inform model owners and their sponsors (senior and executive management) of the expected level of care and documentation. Well-written model risk management policies can also serve to educate all stakeholders about their contribution to model risk management success.
Documentation: The Heart of Good MRM
Documentation associated with a model and the related modeling process is key to successful model risk management. Sufficient documentation is one of the fundamental challenges we see with existing models vs. newly developed corporate model risk management policies. Having substantive documentation—and keeping it current—is critical.
However, documentation historically has had the potential to take a back seat to model managers' other priorities. But this is quickly being reprioritized as model risk management experts and validators increasingly rely on documentation as a key component of model governance. This documentation not only signifies the care and effort taken to properly manage a model, but also provides the benchmark for comparison and validation/effective challenge.
Elements of Good Model Documentation:
→ Convey the purpose of the model to stakeholders, along with its intended use. This is especially important for stakeholders who may not be experts in the particular discipline; e.g., outside directors.
Setting forth the model’s purpose is often addressed within an executive summary. The summary should:
- Include a high-level description of the model
- Highlight key aspects, drivers, or assumptions related to the model
- Specify the risk rating of the model and the results of the last validation
- Explain how the model has performed
- List any known limitations or risks associated with the model
The narrative should be written in layman’s terms and provide a clear and succinct explanation of the model. This gives the second and third lines of defense a defined purpose against which they may judge the model’s performance.
→ Thoroughly describe the model’s theory and design. Institutions should illustrate:
- Why the model was built and what it does
- The model’s general design and mathematical construct
- Architectural considerations
- Data, variables, assumptions
- Selection process (if obtained outside the bank)
These are the basics. Yet organizations often struggle with this aspect of documentation because many legacy models were never documented.
→ Invest considerable effort in data, assumptions, and output documentation (the “nitty-gritty” of the model). Documentation should include:
- A listing of all data and how it is obtained, prepared, transformed, reconciled, and applied
- A listing of all assumptions and their relative impact on model results
- How assumptions are developed and supported, reviewed, tested, and approved
- A description of activities related to assumption sensitivity testing
- Assumption overlays or overrides (factors that are applied to assumption logic or outright replacement of quantitative assumptions) and rationale/support for the adjustments, if applicable
This allows the stakeholders to appreciate that results are not absolute and makes it clear that results depend heavily on assumptions.
A critical component of good model management practices (and documentation) relates to ongoing performance monitoring. Stakeholders should understand the efforts to confirm the regularity and protocols to address exception handling.
→ Thoroughly document everything related to model governance. Be sure to represent every respective role and responsibility, including the various levels of review and oversight, change control, changelogs, etc.
Don't forget the varying technical expertise of your stakeholders—organize your documentation with an emphasis on readability. You might even consider a glossary of terms—don't assume everyone knows all the technical language.
Additionally, write your documentation with a thought towards manageable change—because change will come. This could include the use of an appendix section for details that may be adjusted with regularity.
Lastly, take advantage of electronic mapping. Publish your documentation with an interactive table of contents, hyperlinks to appendices, etc. Easy navigation through your documentation will go a long way in helping your stakeholders use what 'you've built.
Moving Forward
Don't forget to practice what you preach. It's one thing to document a "desired" process – it's another thing to actually follow through and do what you say you are going to do. Make sure whatever you document represents what you are actually doing in practice, and there are demonstrated/verifiable steps. The worst thing an institution could do is to document a process and not follow it. Doing so is an instant recipe for regulatory scrutiny.
Regardless of size, examiners expect institutions to adopt more substantive model risk management practices. At the heart of this heightened activity is good documentation. Good documentation leads to better, high-performing models with improved development and testing disciplines, increased transparency and understanding, and an ability to leverage the strengths of other good modeling practices.
In addition, good documentation has a direct benefit to the bottom line as development, validation, and remediation costs decrease, potential operational risks reduce, and more confident stakeholders make more proactive strategic decisions.
Learn more about our model risk management services.
ABOUT THE AUTHOR
Michael R. Guglielmo is a Managing Director at Darling Consulting Group. With over 30 years of experience in strategic risk management, Mike has provided technical and strategic consulting to a diverse group of financial institutions. Mike is also a frequent author and top-rated speaker on a variety of balance sheet and model risk management and operational risk management topics.
Contact Michael Guglielmo: mguglielmo@darlingconsulting.com or 508-843-8135 to learn how DCG can help you navigate through the new era of elevated MRM and data governance.
© 2022 Darling Consulting Group, Inc.
Comments