> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/xsoar-6-administrator-guide/6.14/onboard-cortex-xsoar/elasticsearch/troubleshoot-elasticsearch.md).
# Troubleshoot Elasticsearch
Your Elasticsearch deployment can have issues with feed ingestion, memory, or general functionality.
{% hint style="info" %}
### Note
PANW support will provide customer help to the best of their ability. You should consider vendor support per need.
{% endhint %}
After reviewing the troubleshooting items in the table below, if you need to create a support ticket:
1. Set the log level to debug by going to Settings → About → **Troubleshooting**.
2. Reproduce the issue and [download server log bundles](/xsoar-6-administrator-guide/6.14/customize-cortex-xsoar/customize-and-configure-cortex-xsoar/logs/create-a-log-bundle.md).
(High Availability) For High Availability deployments, you only need to download the server log bundle once, as it gathers logs from all online application servers. If your High Availability servers are behind a load balancer and the log bundle times out, increase the timeout on your load balancer to five minutes.
3. Attach the logs to the support ticket.
**General issues**
| Issue | Description | Recommendation |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **limit of total fields (X) in index has been exceeded** | Mapping in Elasticsearch exceeded maximum configured field capacity. | Use totalFields under the Elasticsearch configuration in your demisto.conf to set matching index to a custom total fields. By default, common-incident, common-indicator, and common-evidence are set to 2000. You can also set the total limit on a selected index using:
PUT \/\_settings { "index.mapping.total\_fields.limit": 2000 }
|
| **field expansion matches too many fields, limit: X, got: X+Y** | The number of fields a query can target exceeded the limit of X. | Set **`indices.query.bool.max_clause_count`** in **`elasticsearch.yml`** file to a higher value and restart Elasticsearch. |
| request to elasticsearch exceeded maximum size, increase
'http.max\_content\_
length'
in your elasticsearch.yml to allow larger requests
or
(413) request entity too large
| A request to Elasticsearch exceeded the maximum size of http.max\_content\_
length.
| If the request is a bulk save, decrease the **`innerBatchSize`** bulk size that is set to 250 by default. Alternatively, increase the cluster **`http.max_content_length`** settings in **`elasticsearch.yml`** to increase the maximum limit for Elasticsearch received requests. |
| **too many requests to elasticsearch** | | 1. Verify the Elasticsearch cluster state is healthy (green) and all nodes are optimized and balanced. If the too many requests error is from a specific node, it might be related to node optimization.
2. Verify the limit on the number of open file descriptors is sufficient to handle the number of parallel requests.
3. If the error occurs during batch update operations, decrease the base batch size used by Cortex XSOAR when sending requests to Elasticsearch. Edit innerBatchSize under elasticsearch in your demisto.conf file and restart Cortex XSOAR. See the full list of Elasticsearch configurations.
4. If the error occurs when using the migration to Elasticsearch tool, rerun the migration with the -elastic-batch-size flag to lower the batch request size. See the full list of migration flags.
|
| **unable to search on entries** | By default, indexing entries content is disabled for performance reasons. | Set the db.index.entry.disable server configuration to false to enable indexing.
You can also index notes, changes and evidences for searches only using the granular.index.entries: 1 server configuration together with db.index.entry.disable: false.
Note
After making these changes, you will be required to wait for a new month to apply the new mapping. To apply the change to the existing entries you will need to reindex the common-entry\_\* indices.
\[400] Failed with error \[1:417] \[bool] failed to parse field \[must]. Other reasons: \[\[{x\_content\_parse\_
exception \[1:417] \[bool] failed to parse field \[must]}]]
| Queries with multiple **`or`** conditions, such as **`"id:1 or id:2 or id:3 or id:4 or id:5 or id:6 or id:7 or id:8 or id:9 or id:10"`** may fail when using Elasticsearch v7.14 or later. | Avoid using **`or`** by providing a value list **`"id:(1 2 3...)"`** |
| **cannot restore index \[.geoip\_databases] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name** | When using Elasticsearch v7.14 or later, you may encounter failures when restoring from snapshots. | Add **`ingest.geoip.downloader.enabled: false`** to your Elasticsearch configuration file. |
| ReleasableBytesStream
Output
cannot hold more than 2GB of data. Other reasons:
\[\[{illegal\_argument\_
exception
ReleasableBytesStream
Output
cannot hold more than 2GB of data}]] ) \[error '\[400] Failed with error:
ReleasableBytesStream
Output
cannot hold more than 2GB of data
| If there are insufficient shards, Elasticsearch’s circuit breaker limit may be reached due to the search load. | Increase the number of shards. For example, if you have a three-data-nodes cluster, you should have at least two replicas for each active shard, making the data available across all nodes. We also recommend using [nodes stats](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html) to verify the nodes are balanced for read and write operations. |
| **Data too large, data for \[\] would be \[xxxx/xxxx], which is larger than the limit of \[xxxxx/xxxxx]** | Elasticsearch's circuit breaker limit is reached due to the load of indexing operations. | 1. Verify you have sufficient JVM. The Elasticsearch default setting is significantly lower than required.
2. Verify you have implemented Cortex XSOAR best practices for Elasticsearch and set up a sufficient number of primary shards for the heavily used indices.
3. Verify the cluster is balanced correctly by using nodes stats. Identify unbalanced nodes that store most or all primary shards, causing node write to queue and causing decreased performance on all shards in the node. In this case, we recommend moving some primary shards to a less busy node and/or adding additional primary shards to ensure the nodes are balancing their resource usage (i.e. CPU, memory and IO).
|
| **HTTP 504 gateway request timeouts** | HTTP requests are timing out and preventing playbook data from loading in the browser. | If your high availability servers are behind a load balancer, increase the timeout on your load balancer to 300s. |
**Memory issues**
| Issue | Description | Recommendation |
| ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Insufficient JVM memory** | The default JVM memory is 1 GB. In production environments, this might be insufficient. | Increase the JVM memory
|
| **Insufficient term query size** | The term query size is used by the bulk edit. The default term query size is 65536 and may be insufficient. | Increase the [term query size](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-max-terms-count). |
| **Insufficient bulk size** | The [bulk size](https://www.elastic.co/guide/en/elasticsearch/reference/current/circuit-breaker.html) depends on the available JVM memory and affects the amount of data that Cortex XSOAR can send and process in Elasticsearch. | |
| **Heap size** | | The recommended maximum [heap size](https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html) is 50% of the entire server, as long as the other 50% is free. |
| **Performance issues due to swapping enabled** | | [Disable swapping in Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/master/setup-configuration-memory.html) to improve performance. |
**Feed Ingestion Issues**
| Issue | Description | Recommendation |
| ------------------ | ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Stack overflow** | In some cases, complex search queries cause Elasticsearch to fail on stack overflow. | Use the following search query syntax: field:(a,b,c …), which is based on a clause count. In this example, a, b, and c each represent a clause.
To determine how many clauses a query can contain, set the maximum clause count and the maximum total field count in the elasticsearch.yml file.
Maximum clause count
Key:
For ES 6.0 and later the key is index.query.bool.max\_clause\_count.
For ES 5.x and earlier the key is indices.query.bool.max\_clause\_count.
Default: 1,024. You can increase the value.
Maximum total field count
Key: index.mapping.total\_fields.limit
Default: 1,000. You can increase the value.
|
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://cortex-docs.paloaltonetworks.com/xsoar-6-administrator-guide/6.14/onboard-cortex-xsoar/elasticsearch/troubleshoot-elasticsearch.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.