Model validation: An essential part of any analytics program
You’ve started analyzing your data. You’re seeing results. What’s the next step?
Hopefully you answered model validation, or verifying if your models are performing as expected. Whether your credit union is building models internally or outsourcing your analytics projects, it is becoming increasingly important to validate the predictive accuracy of your models. In other words, are the models’ predictions close to what actually happens?
Model validation is about identifying discrepancies and shortcomings in both design and performance. It also helps credit unions evaluate the methodology used by the model, which is important, as the rules and regulations surrounding data collection will likely tighten in the near future.
Most importantly, model validation provides a high level of comfort to decision makers who should be more focused on strategy than wondering if their decisions are based on accurate predictions.
Model validation can be a heavy lift, so it’s best to break the process down into five manageable steps.
- Determine best practices. As you look at revamping or creating your model validation process, there’s no reason to recreate the wheel. Find out what others are doing well and not so well.
- Develop a workflow chart. The validation process can be complex, so it is important to define the step-by-step process. Be sure to identify the individuals who will participate and what roles they will play.
- Establish process and reporting requirements. The model review process should call for the identification of potential sources of error or bias in data, theory and assumptions, as well as standards for analyzing and reporting results prior to approval for deployment.
- Report findings. The report should include the method of validation, an evaluation of the ongoing effectiveness of the model, as well as the weaknesses and potential improvements that can be implemented into the model design.
- Create a validation calendar. If you don’t schedule the time to evaluate a model, it won’t happen. Your most valuable models should be checked at least annually. Those for special projects should be reviewed every two or three years.
Even if you aren’t doing your own model validation internally, you should still understand the process your analytics partner will undergo. Make sure there is a report or manual you can review that outlines the process. Be clear on how often the model is validated.
Model validation can seem like getting into the weeds of the analytics process. But the reality is, if your models don’t work, then your data analytics efforts are essentially for not. Taking time to evaluate the effectiveness of models is essential to the continued health of your credit unions analytics effort, and ultimately, its bottom line.
