Dealing with Technological Downtime
One Friday in 2005, the ECR system experienced a massive system shutdown when a storage array failed. Upon beginning work toward recovery, staff found that the emergency backup database system required modification to work properly. Not only had the primary system failed - the backup system failed as well. This was among the least expected and worst possible scenarios.
Clerk's Office staff worked tirelessly to address both the initial failure as well as the failure of the backup system. It took two weeks to stabilize the system. Staff worked with vendors to replace necessary SQL server binaries and recover the master database, as well as the ECR and FileNet databases from backups. Challenges went beyond the technological as well: customer service during the downtime was drastically affected, and Clerk's Office processes were shaken by the time and costs of catching up following the system’s restoration.
Following this experience, the Clerk's Office had the opportunity to re-examine their work processes on several levels. Technology can and will break, and systems were established that more effectively prepared for that possibility. Clerk's Office staff re-worked their notification mechanisms for internal and external users. They updated their emergency filing processes, and documented and tested those plans. Internal workflows were strengthened and clarified. Customer service processes were also updated, to better address customer issues that could arise in the event of system downtime.