Disaster Recovery

What if your environment gets pwned? Or you lose your local hypervisor and everything running on it? Or you lose all your precious configurations due to a human error or some glitch?

The Black Swan

---

Breakdown

Personnel

If something should happen, it might be a good idea to gather a list of contacts that can be reached in such event. Also, making sure proper service levels and channels are set in place. One should never rely on goodwill or “I know this one guy who’s always willing to help if something happens” – because, what if they can’t be reached?

Also consider that there should always be someone on duty, ready to pick up the phone and ready to take action if needed. They also should be made aware of critical systems and who are the ones reachable in such event.

Prevent

Evaluate your environment design regularly and make the necessary changes in a controlled manner. It is cheaper than the alternative.

Patch – make sure to implement security updates for everything asap.

Leverage modern authentication (passwordless, FIDO2) to rule out the possibility of leaking passwords.

Mitigate

Is it possible to narrow down the issue? Is it possible to shut down the exposed system and find an alternative way to allow people to complete their tasks?

• If the external access gateway is lost, is it possible to summon the personnel to the office or use some other way to connect?
• If a business critical system is down, is there an alternative way to complete work? Paper and pen even? How long can you run your business in such case until you need to stop operations?
• If a (D)DoS attack hits your environment and either your internet connection capacity reaches its limit, firewall can’t handle the traffic, the published remote connection gateway, web servers, etc… go down, is it possible to steer the traffic via other route or is the only option to drop the inbound connections for example? How can you achieve the necessary mitigation with minimal impact to production use, when should you consider doing something like this and by who?

Remediate / recover

Preparation is the key here. Are you running backups? Have you tested restoring the backups? How? Who?

If an appliance, hardware or virtual, gets pwned, is it possible to do a factory reset (for hardware), get a replacement hardware box or with virtual, is it possible to create a new VM? Then patch it and just restore the safe configuration in that box?

Also, in case of cyber attack it is a good idea to:

• Reset all the passwords related to that box operations
• Re-key all certificates
• If the box is handling passwords (not just acting as a OIDC/OAuth2 or SAML2 Service Provider), enforce password resets for the users for that environment

Don’t mess up the crime scene (if applicable)

If there’s a suspicion of crime, it’s really important to preserve the evidence for forensics purposes. This might have a huge financial impact if your business takes a hit and depending on your possible cyber insurance policy, you might need to take in to account what limitations it might have.

For example, it might be a very bad idea just to power down some exposed appliance if the malware is running in its memory. The “bad idea” is likely different for every system (web server, database, file storage, user directory…).

References

Here are some references you can check:

• NetScaler – CTX694799 “Steps to Take if NetScaler ADC is Suspected to be Compromised”.
• DaaS – https://docs.citrix.com/en-us/citrix-daas/backup-migrate-configuration/backup-restore-configuration
• XenServer – https://docs.xenserver.com/en-us/xenserver/9/dr
• Active Directory – https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/manage/forest-recovery-guide/ad-forest-recovery-procedures
• Active Directory Certificate Services (ADCS) – https://techcommunity.microsoft.com/blog/microsoftsecurityexperts/recover-an-adcs-platform-from-compromise/4120889

What next?

---