The planning and execution of disaster recovery
The Plan
Business continuity planning is a lot like homeowners insurance: you prepare, plan and pay the premiums in the hope of never having to use the benefits.
Every professional credit union technology leader can appreciate the value of developing a written plan for disaster recovery. Most mature operations perform a test of the plan annually by simulating a localized disaster event in a tabletop exercise during which what-if scenarios are played out and assumptions for contingencies are vetted. However, most of these same leaders have never experienced a true disaster that requires their plans to be put to the test.
PSCU maintains data center and processing operations at its headquarters in St. Petersburg, Florida. While the greater Tampa Bay region maintains a perennial spot on the list of top metropolitan areas likely to feel the effects of a major hurricane, the area had not experienced a direct hit from a strong hurricane since 1921. In preparation for a potential disaster, PSCU has spent years building out its service, fraud, call center and technical operations in its Phoenix, Arizona, facility in order to achieve full redundancy across both processing centers.
Test and Test Again
In the summer of 2016, PSCU ran its first true production test operating completely from a single center for several days – first in Arizona and then in Florida. This production test was repeated during the early summer months of 2017 for an extended period, including during peak processing time frames. These tests resulted in a number of valuable insights and key takeaways, which the company diligently addressed and then tested, optimized and retested.
Executing the Plan
Former world heavyweight champion Mike Tyson once said, “Everyone has a plan until they get punched in the mouth.”
In late August of this year, a tropical wave off the western coast of Africa was formally upgraded to Hurricane Irma. As the storm intensified and passed through the Caribbean, it was upgraded to a Category 5, becoming one of the largest and most intense hurricanes ever recorded in the Atlantic.
As the storm turned north and began to track up the west coast of Florida, PSCU invoked its business continuity plan and upgraded its status to the highest level of vigilance. As planned, employee attendance policies were altered, emergency supplies were distributed, and operations and technical processing began shifting from Tampa to Phoenix. By the time the eye of the storm hit the region, PSCU was handling its full service load in Arizona, and Florida operations were essentially dark.
Ultimately, the Tampa region was spared from the worst of the impact. Despite this good fortune, the area was lashed with 100 mph winds, and much of the region was without power for more than a week. The final tally pegs total damage estimates from the storm at roughly $63 billion, making it the fourth costliest storm on record.
Final Analysis
Looking back, PSCU’s planning, strategic investments and execution allowed the focus of disaster efforts to be primarily on supporting employees, Member-Owner credit unions and their members. While processing was uninterrupted throughout the event, families, schools and surrounding communities were hit hard, and the cleanup is still ongoing.
As the recent events in Texas, Puerto Rico and California illustrate, an academic planning exercise can turn into an actual disaster execution exercise in the blink of an eye. In these times, nothing beats a well-considered, tested and executed action plan.
