Ctrlk
BlogSupport
For the complete documentation index, see llms.txt. This page is also available as Markdown.
Cortex XSOAR6.12EoLCortex XSOAR 6.x

Configure the Live Backup Environment

Configure a live backup environment in Cortex XSOAR 6.12.

Live Backup enables you to mirror your production server to a backup server, and in disaster recovery scenarios to easily convert your backup server to be the production server.

When using Cortex XSOAR with Elasticsearch, Live Backup is not available. To back up or restore the contents of your Elasticsearch database, follow the instructions for Disaster Recovery for Elasticsearch. Alternatively, you can also implement a full high availability solution.

Caution

  • Before you start, ensure that you save the disaster recovery configurations before you copy all files.

  • If the main server has a custom certificate with password, you need to create a one-time-configuration (OTC) file (see Step 8).

  • These instructions do NOT apply to multi-tenant deployments. Instead, follow the Configure Live Backup instructions in the multi-tenant guide.

  1. On the production server, enable live backup.

    1. Go to Settings → About → Troubleshooting → Server Configuration.

    2. Verify that the External Host Name is correct.

    3. Click Add Server Configuration.

    4. Add the following key and value.

      Key
      Value

      ui.livebackup

      true

    5. Go to Advanced → Backups and in the Live Backup field, select ON.

    6. Add the following backup server parameters.

      Parameters
      Value

      Hostname/IP Address

      Backup server IP address or Host name (without https:// prefix).

      Port

      Default is 443.

      Trust server certificate (unsecured)

      ON: certificates are not checked. OFF: certificates are checked.

      Use proxy

      Select whether to use a proxy.

  2. On the backup machine (with a different host name or IP address), install Cortex XSOAR.

    1. sudo ./demistoserver-xxxx.sh -- -dr -do-not-start-server

    2. Verify that the backup server is accessible from the production server through port 443 (or any other port configured as a listening port). Ensure that there are no firewalls that might drop communication.

  3. On the production server, stop the Cortex XSOAR server:

    sudo service demisto stop

  4. On the production server, create a tarball file of the necessary files and folders on the production server and copy it to the backup server.

    Note

    This process may take several hours, depending on your server specs and the amount of data that needs to be copied.

    1. Ensure that all files and folders located in /var/lib/demisto have demisto:demisto ownership:

      chown -R demisto:demisto /var/lib/demisto

    2. Create the tarball file:

      tar --ignore-failed-read -pczf demistoBackup.tgz /var/lib/demisto/data /var/lib/demisto/artifacts /var/lib/demisto/attachments /var/lib/demisto/images /var/lib/demisto/systemTools /var/lib/demisto/d2_server.key /usr/local/demisto/cert* /usr/local/demisto/demisto.lic

      Sometimes the demisto.lic file is located in /var/lib/demisto/demisto.lic rather than /usr/local/demisto/demisto.lic. If so, change the directory in the command.

      If you have not set up a D2 server, you can remove /var/lib/demisto/d2_server.key.

    3. Verify the integrity of the tar file:

      md5sum demistoBackup.tgz

    4. Print the contents of the tar file to a text file:

      tar -tvf demistoBackup.tgz > demistoBackup.txt

      Do not delete the text file.

    5. Transfer the tarball file (demistoBackup.tgz) to the backup server, using your preferred tool such as scp:

      # scp demistoBackup.tgz root@<yourBackupServerIPortHostname>:/root

  5. On the backup server, check the MD5 Checksum and compare it to the original file to verify the tar file is 100% valid:

    md5sum demistoBackup.tgz

    The MD5 sum is displayed. Compare this value against the MD5 sum saved in demistoBackup.txt in Step 4.

  6. On the backup server, extract the backup tarball file (original file permissions and ownership are preserved):

    sudo tar -C / -xzpvf demistoBackup.tgz

  7. Ensure all the copied files and folders have demisto:demisto ownership.

  8. If the main server has a custom certificate with password, you need to add the key password to the one-time-configuration (OTC) file located in /var/lib/demisto/otc.conf.json. After the file is saved and the Cortex XSOAR server is restarted, the OTC file is automatically deleted. Add the following content to the OTC file: {"keypass":"certpassword"}. The otc.conf.json must have permissions for demisto:demisto.

  9. Start the backup server:

    sudo service demisto start

  10. Start the production server:

    sudo service demisto start

    If the procedure is successful, Live Backup is ON.

    If the server is active, Cortex XSOAR appears as usual when you connect. You can Transition an Active Server to Standby Mode or Transition a Standby Server to Active Mode.

Configure Live Backup for Multiple SAMLs

These steps describe how to configure live backup for multiple SAMLs by configuring SAML integrations for each server.

  1. Verify the External Host Name value matches the URL of the production server by going to Settings → ABOUT → Troubleshooting → Server Configuration.

  2. On the production server:

    1. Go to Settings → INTEGRATIONS → Instances and search for SAML 2.0.

    2. Create a SAML integration by clicking Add Instance.

    3. Add the required parameters. For more information about setting up a SAML integration, see Authenticate Users with SAML 2.0.

      Ensure that Service Provider Entity ID parameter matches the URL of the production server. This is the only valid value. For example, https://prod.demisto.com/saml.

    4. Verify that you can access the production server using SSO authentication.

  3. On the disaster recovery (DR) server:

    1. Create a SAML integration by repeating step 2.

      Ensure that the Service Provider Entity ID parameter matches the URL of the DR server. This is the only valid value. For example, https://dr.demisto.com/saml.

    2. Verify that you can access the production server.

  4. (Optional) For Dev/Test environments, switch to the DR server and test the SSO login.

DR Scenario: Testing the DR Environment

Before you make the backup server the new production server, ensure the original production server is down (not live). If you make the backup server the new production server while the original production server is still live, you can experience significant issues.

After completing your test of the failover scenario (backup environment), you need to revert the server to its original state, which you can do through the UI or by modifying the configuration file.

Note

(Multi-tenant) - When using Live Backup in a multi-tenant deployment, each host machine has a backup server. When testing the disaster recovery environment, follow these steps to test each host machine separately, including the main account machine.

  1. If you want to use Cortex XSOAR to backup the server, do the following:

    1. On the live production server, select Settings → Advanced → Backups → Switch Hosts.

      (Multi-tenant) - For a multi-tenant deployment, go to Settings → Account Management → Hosts. If you want to switch for the main host instead, go to Settings → Account Management → Main Hosts. Select the desired host, click the Live Backup button, and then Switch Hosts.

    2. When prompted, complete the online Switch Hosts instructions.

      Ensure that the production server is not live.

      (Multi-tenant) - Ensure that the production server for this machine is not live.

    3. Go to the backup server and follow the on-screen instructions to make the backup server the production server.

    4. In the backup server environment, go to Settings → About → Troubleshooting.

    5. For the External Host Name key, update the value to the host name of the backup server (the new production server).

      The backup server is unaware of its external host name.

      After a successful switch, the backup server is now live.

  2. If you want to use a configuration file to test the DR environment, do the following:

    1. On the production server, stop the server:

      sudo service demisto stop

    2. Open the /etc/demisto.conf file on the production server. If the Server.dr.enabled property is in the file, change it to true. If it is not in the file, add Server.dr.enabled and set it to true.

    3. Start the server and verify it is in disaster recovery mode. You should see This is currently the backup server.

    4. On the backup server, stop the server:

      sudo service demisto stop

    5. Open the /etc/demisto.conf file on the backup server. If the Server.dr.enabled property is in the file, change it to false. If it is not in the file, add Server.dr.enabled and set it to false.

    6. Start the backup server.

      The original production server is now the backup server and the original backup server is now the production server. Access the new production server using its IP address to check that the server is up and running.

  3. To revert to the original settings, repeat the steps above.

    When following the instructions, remember the backup server is now the production server and the production server is now the backup server.

DR Scenario: Unrecoverable Active Server Failure

In the event of an active server failure where the server cannot be restored (flooding, fire, meteor impact, hardware failure, etc.), follow these steps to convert the backup server to the new production server and then configure a new backup server.

Note

(Multi-tenant) - In a multi-tenant deployment, each machine (main account and host machines) has its own backup server. You can switch from the production server to the backup server for any or all of the machines in your deployment.

Before you start you need to Configure the Live Backup Environment.

  1. On the standby server, follow the steps in Transition a Standby Server to Active Mode.

  2. If your analysts use a single, pivoting host name to connect to the active Cortex XSOAR server, update your DNS record to re-point your Cortex XSOAR server host name to the now active server. For more information about host names, see Host Names, DNS, and Disaster Recovery.

  3. If using engines, confirm that they are connected in Settings → Integrations → Engines.

    If they have not reconnected and you have confirmed that network connectivity is good between the engine and the now active (previously backup) server (i.e., it is reachable on TCP 443 or the port you have configured), then follow the guidance in Host Names, DNS, and Disaster Recovery.

  4. Obtain a new server environment according to your requirements.

    Do not install Cortex XSOAR until step 5.

  5. Follow the procedure for Configure the Live Backup Environment using your now-active server as the primary host, and copying its files and data to the newly-built Cortex XSOAR server. Confirm that Live Backup is working.

  6. If appropriate for your environment (depends on whether you want to remain on the present active node), transition the active node over to the newly-built host by following the procedure Transition an Active Server to Standby Mode and confirming that Live Backup is again operational.

  7. If applicable, follow Step 3 to reconfirm that engines are connected.

  8. Re-point your shared DNS record, if applicable, back to the primary Cortex XSOAR server and have analysts reconnect.

  9. Confirm that Cortex XSOAR is working by confirming that your integrations are working properly, incidents are being created normally, and that analysts can login and work normally in Cortex XSOAR.

DR Scenario: Unrecoverable Standby Server Failure

In the event of an unrecoverable standby server failure (flooding, fire, meteor impact, hardware failure, etc.), follow these steps.

Before you begin, ensure that you Configure the Live Backup Environment.

  1. Obtain the new server that will serve as your new standby server according to your requirements.

    Do not install until step 2.

  2. Follow the procedure for Configure the Live Backup Environment using the newly-active server as the primary host, and by copying its files and data to the newly-built Cortex XSOAR server. Confirm that Live Backup is working.

  3. Test the new server by following the steps in DR Scenario: Testing the DR Environment.

PreviousHost Names, DNS, and Disaster Recovery
NextTransition an Active Server to Standby Mode

Last updated

Was this helpful?