> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/xsoar-6-administrator-guide/6.14/onboard-cortex-xsoar/elasticsearch/archive-data-with-elasticsearch.md). # Archive Data with Elasticsearch Cortex XSOAR supports archiving of partitioned data. Partitioned data is stored in indices on a monthly basis for easy archiving and accessibility. To free up disk space, you can create Elasticsearch snapshots of the relevant indices and then delete the indices. **Manual** To manually archive older data we recommend deleting all indices for a specific month after creating a snapshot for that month. For example, to delete all January 2020 data, use the following API call: **`DELETE *dmst-*_202001`**. This can be safely done at any given time without shutting down the Cortex XSOAR service. To restore archived data, follow the [restore](/xsoar-6-administrator-guide/6.14/onboard-cortex-xsoar/elasticsearch/disaster-recovery-for-elasticsearch.md#UUID-a9770752-cb64-9f6b-f131-46e67bd31818) instructions for Elasticsearch backups. **Automated** Elasticsearch supports index lifecycle management through the ILM API, to automatically manage indices retention and optimize old indices. You can [set up an ILM policy](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html) through the Elasticsearch API or the Elasticsearch UI. OpenSearch also provides an automatic archiving option, through index state management. You can manage an [ISM policy](https://opensearch.org/docs/latest/im-plugin/ism/index/) through the OpenSearch API. You can configure a default ILM/ISM policy for all partitioned indices, or for a specific index type. Cortex XSOAR does not currently support rollover operations. It is not necessary to define all phases of the policy. If a phase is not defined, the next available phase is processed. **Define ILM (Elasticsearch clusters)** 1. Create or update the lifecycle policy. ```programlisting PUT _ilm/policy/ ``` 2. Update the index with the lifecycle name.

Note

The index_name can be a wildcard. For example, PUT *-dmst-*202203/_settings defines the policy on all partitioned indices for March 2022. Do NOT use wildcards for non partitioned indices.

```programlisting PUT /_settings { "settings": { "index.lifecycle.name": } } ``` **Define ISM (OpenSearch clusters)** 1. Create or update the lifecycle policy. * Create the lifecycle policy. ```programlisting PUT _plugins/_ism/policies/ ``` * Update the lifecycle policy. Retrieve the **`_seq_no`** and **`_primary_term`** for the existing policy: ```programlisting GET _plugins/_ism/policies/ ``` Use the **`_seq_no`** and **`_primary_term`** values retrieved in the previous step to update the lifecycle policy. ```programlisting PUT _plugins/_ism/policies/policy_1?if_seq_no=[seq_no]&if_primary_term=[primary_term] ``` 2. Update index settings with policy\_id.

Note

The index_name can be a wildcard. For example, PUT *-dmst-*202203/_settings defines the policy on all partitioned indices for March 2022. Do NOT use wildcards for non partitioned indices.

```programlisting PUT /_settings { "settings": { "plugins": { "index_state_management": { "policy_id": [policy_name] } } } } ``` 3. Add a policy to an index. ```programlisting POST _plugins/_ism/add/[index_name] { "policy_id": [policy name] } ``` **Examples** In the following examples, we use hot, warm and delete phases, with the allocate, forcemerge and delete actions. 1. Each index stays at the hot phase for 90 days, and then moves to the warm stage. 2. In the warm stage, the number\_of\_replicas and total\_shards\_per\_node is reduced to 1, and forcemerge is executed. 3. After one year the index is deleted. **Elasticsearch ILM example** ```programlisting PUT _ilm/policy/default_policy { "policy": { "_meta": { "description": "This is a default policy", "project": { "name": "dmst", "department": "dev" } }, "phases": { "warm": { "min_age": "90d", "actions": { "allocate": { "number_of_replicas": 1, "total_shards_per_node": "1" }, "forcemerge": { "max_num_segments": 1 } } }, "delete": { "min_age": "365d", "actions": { "delete": {} } } } } } ``` **OpenSearch ISM example** ```programlisting PUT _plugins/_ism/policies/default_policy { "policy": { "description": "This is a default policy", "default_state": "hot", "states": [ { "name": "hot", "actions": [], "transitions": [ { "state_name": "warm", "conditions": { "min_index_age": "90d" } } ] }, { "name": "warm", "actions": [ { "replica_count": { "number_of_replicas": 1 } } ], "transitions": [ { "state_name": "delete", "conditions": { "min_index_age": "365d" } } ] }, { "name": "delete", "actions": [ { "delete": {} } ], "transitions": [] } ] } } ``` --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://cortex-docs.paloaltonetworks.com/xsoar-6-administrator-guide/6.14/onboard-cortex-xsoar/elasticsearch/archive-data-with-elasticsearch.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.