IaC and DRP Development for Legacy Architecture

A client whose key business processes were tied to databases, software modules, and other core architecture components that had lost support several years ago approached us. All servers were configured manually, and changes were made by dozens of different teams without detailed project documentation. As a result of an internal audit, it was decided to eliminate technical debt, as the failure of these components meant a complete shutdown of the client's business for 3 weeks or more. The forecasts for full recovery were negative.

The client's primary need was a complete description of each component of their architecture with the creation of documentation, the development of code for deploying all nodes, and the development of a Disaster Recovery Plan.

The most challenging part of this project was gathering information. In preparation for the upcoming work, we conducted a series of interviews with each employee to collect as much data as possible on the systems they had personally encountered or heard about from colleagues who had left. We processed, systematized, and described all data in a standard format. Another problem was that the software components' code, configuration files, and other data were stored in 27 different repositories. We organized a search across all accumulated data, creating an index based on the open on-premise solution Sourcegraph. The source codes were a mix of Golang, Python, bash, PHP, and numerous plain-text configuration files. Next, we began the painstaking work of separating individual components from a highly interconnected architecture. The work was carried out in several stages:

• Description of a new architectural component in Ansible and Terraform
• Deployment of the node in a test environment, conducting integration testing, and equivalence tests to manually configured
• Full resetup of the component using IaC.

For components with compatibility issues and lacking source codes, we, in agreement with the client, rewrote them onto a fresher technological stack. Some critical components were left to operate as a "black box," ensuring backup, automatic deployment, and documentation. In the final iteration of our work, we successfully tested the deployment of the client's core architecture on two independent sites. During integration testing, we conducted successful partial and full switching of productive load to the backup. The result of our work was a developed DRP plan with simple, clear documentation and deployment of the entire infrastructure in a few requests through the use of Terraform and Ansible. We made modifications to the client's architecture, achieving the effect of graceful degradation in the event of a complete core failure. In this situation, clients experience a partial reduction in available service functionality, but overall operability is maintained. During this time, within 20 hours, backup infrastructure is deployed on backup sites. This allowed for zero costs for backup infrastructure, which is deployed from scratch only in case of a total failure.