Migrate a Single Instance for High Availability

Migrate a single instance to high availability in Cortex XSOAR 6.12.

To migrate a single instance from BoltDB to Elasticsearch with High Availability, there are two steps. First use the Cortex XSOAR migration tool to migrate the database. Then set up additional app servers to achieve high availability.

Migration

The migration tool migrates Cortex XSOAR objects to an Elasticsearch database. When you run the migration tool, the contents of the Cortex XSOAR database are read, and a corresponding object is created in Elasticsearch.

In the BoltDB, data related to incidents and indicators is stored by month in partitions. To minimize downtime during migration, we recommend creating a copy of the database and migrating data that is older than three months from the copied database. This enables you to continue to work in your current environment. When the initial migration completes, migrate the remaining months.

Note

You cannot run more than one migration tool process at a time.

All commands are run from the Cortex XSOAR server machine.

If you are upgrading from Cortex XSOAR v6.0 and you have indicators and audits stored in Elasticsearch in Cortex XSOAR v6.0, upgrade from Cortex XSOAR v6.0 before the migration. Do NOT start the server. Then follow the migration instructions below.

Before you begin, ensure the following:

• You have an active instance of Cortex XSOAR.
• All app servers can communicate with Elasticsearch over port 9200.
• All app servers have network access to each other over port 443.

1. Copy the demisto.lic file from /usr/local/demisto directory to the /var/lib/demisto directory.
2. Add the following entry in the demisto.conf file:

   license.file.path:"/var/lib/demisto"
3. Follow migration instructions:

   • If you did not previously store indicators and audits in Elasticsearch, follow the migration instructions for single server deployments.
   • If you used Elasticsearch to store indicators and audits in Cortex XSOAR v6.0 prior to upgrade, ensure the demisto.conf file is up-to-date with the ES object and then Migrate an Existing Elasticsearch Deployment.

Migration Tool Flags

Flag	Type	Description	Required
accounts (multi-tenant only)	String	A comma-separated list of accounts to migrate. If not specified, all accounts are migrated.	Optional
config-path	String	The path to the configuration file for the server. Default: /etc/demisto.conf.	Optional
db-path	String	The path to the database directory. Default: /var/lib/demisto.	Optional
elastic-batch-size	Integer	The number of indicators per batch to write to Elasticsearch indexes. Default: 700.	Optional
elastic-index-prefix	String	The index prefix used in Elasticsearch.	Optional
elastic-key	String	The API key to connect to Elasticsearch.	Required (unless a username and password are used)
elastic-password	String	The password to connect to Elasticsearch.	Required (unless API key is used)
elastic-url	String	The URL of your Elasticsearch environment. Default: http://localhost:9200.	Required
elastic-username	String	The username to connect to Elasticsearch.	Required (unless API key is used)
ignore-ids-path	String	The path to the file with the IDs to ignore, per object.	Optional
log-level	String	The log level to display. Default: info.	Optional
logfile	String	The location of the log file.Default: /var/log/demisto/elastic_migration.log	Optional
log-failed-items	Integer	Log individual failed items, either in a single meta file, or file per item failure. Values: 0: Does not log failed items (default). 1: Logs failed items metadata in a single file. 2: Logs each failed item in an individual file under the log folder. For more information, see Troubleshoot Elasticsearch.	Optional
migrate-all	Boolean	By default, the Elasticsearch tool checks existing indexes and migrates only the ones that are new. Using this flag, the Elasticsearch tool migrates all indexes even if they currently exist. This is useful, for example, if there was an error or invalid data that was fixed. When used, the objects-to-migrate and objects-to-ignore flags are ignored. Values: true (default) false	Optional
objects-to-ignore	String	Comma-separated list of objects not to migrate. When the migrate-all flag is used, this flag is ignored.	Optional
objects-to-migrate	String	Comma-separated list of objects to migrate. When the migrate-all flag is used, this flag is ignored.	Optional
partitions	String	Comma-separated list of partitions to migrate. If no partitions are specified, all partitions are migrated.	Optional
partitions-to-ignore	String	Comma-separated list of partitions to exclude.	Optional
previous-results	N/a	Show results of the previous migration.	Optional
skip-existing-indicators	Boolean	Existing indicators are not modified during the migration. Values: true false (default)	Optional
object-max-size	Integer	The maximum size, in megabytes, of objects that will be migrated to Elasticsearch. The default is 100 MB.
retry-large-objects	Boolean	Retry the migration of large objects when re-running the migration tool. With this flag, the entire bucket that contains the skipped large object is migrated again, which may include data that was previously migrated. If new data has been added in Elasticsearch since the earlier migration, this data will be overwritten.
force-main	Boolean	When the partitions flag is used, the -force-main flag forces the migration of the main partition as well as the specific list of partitions.
partitions-back	Integer	Provides an option to migrate X number of partitions back. For example, migrating 3 partitions back migrates the current month and the previous two months. If set to 0 or not used, all partitions are migrated.
only-old-partitions	Boolean	Can only be used with the partitions-back flag. Migrates all partitions other than the most recent partitions specified with partitions-back. For example, -partitions-back 3 -only-old-partitions migrates all partitions besides the current month and the two previous months. Should be used within the same calendar month as the previous migration.
version	N/a	Prints the migration tool version.	Optional
y	N/a	Answers yes to all questions, unless there is an error.	Optional

High Availability

If you are implementing high availability and have not configured your load balancer to provide SSL, ensure that you have the same certificates for all the app servers and that the web certificate contains all the app server and load balancer URLs.

1. When working in a high availability configuration, you must mount /var/lib/demisto/ on a shared file system, to allow application servers to share files such as license, artifacts, and attachments. When migrating an existing file system to a shared file system, move the /var/lib/demisto folder without the /var/lib/demisto/temp subdirectory to the shared directory.
2. The temp directory must be local and not shared. To change the location of the temp directory, edit the folders.temp key in the demisto.conf file.
3. Set the app server’s hostname by editing the demisto.conf file:

   Under the Server object, set the value of the inClusterHostname parameter to the desired hostname. The hostname should be the internal address used for communication with the other app servers.
4. Apply a signed certificate.
5. Install additional app servers.

   Verify that the clocks on the app servers are synchronized with an NTP server.