> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/xsoar-6-administrator-guide/6.14/onboard-cortex-xsoar/engines/troubleshoot-cortex-xsoar-engines.md). # Troubleshoot Cortex XSOAR Engines When troubleshooting Cortex XSOAR engines, access the logs from Settings → Integrations → **Engines** and select the engine from which you want to download the logs. ### Debug Engines The **d1.log** field appears whenever a Cortex XSOAR engine is running. The **d1.log** field contains information necessary for your customer success team to debug any engine-related issue. The field displays any error, as well as noting whether the engine is connected. ![engine\_debug.png](/files/NsAnfPu5ie0nbzXiCXjB) If you encounter issues when connecting to an engine, follow these procedures: * [Troubleshoot Engine Installation](#troubleshoot-engine-installation) * [Troubleshoot Engine Upgrades](#troubleshoot-engine-upgrades) * [Troubleshoot Engine Connectivity](#troubleshoot-engine-connectivity) * [Troubleshoot Integrations Running on Engines](#troubleshoot-integrations-running-on-engines) ### Troubleshoot Engine Installation After installing the engine, check that the XSOAR engine is connected to the main server and that it is running. 1. Go to **Settings** → **INTEGRATIONS** → **Engines** and verify that the engine is connected. ![engines-troubleshooting-connected.png](/files/KKYC0mvUmXTqT0QHBwL6) 2. If the engine is not connected, run the following command on the engine server to check if the engine service is running. **`sudo systemctl status d1`** 3. Access the d1 log on the engine server. **`sudo tail -f /var/log/demisto/d1.log`** * If the engine service wasn’t running, and there’s nothing relevant in the log, run **`journalctl`** on the engine server to understand why the installation failed. * If the engine service is running, review the errors to see if the engine is failing to connect or if there are other issues. (Ignore all errors related to \d2ws, since this is not the same as d1ws.) Most often, the server address is incorrect and you will see an error like this: `error Cannot connect to [wss:///d1ws]: wss:///d1ws: dial tcp: lookup localhost: no such host. . Waiting 3 seconds. Will try until…` In this case, navigate to `/usr/local/demisto/d1.conf` and change the EngineURLs parameter to an address the engine can reach. Check the addresses at the beginning of the upgrade\_engine.sh file and update them to be the same as in the conf file. The addresses should be a comma-separated list.

Note

You can ignore the following error: Cannot create folder ‘/var/lib/demisto’.

The configurations that might affect the upgrade\_engine.sh script are the following variables located at the beginning of the script: * `SERVER_URLS` * `TRUST_ANY_CERT` If you make a change to the baseURLs configuration, you must apply the change in `/usr/local/demisto/d1.conf` AND in `/usr/local/demisto/upgrade_engine.sh` under the SERVER\_URLS var. If you make a change in the `engine.connection.trust_any_certificate` configuration, you must apply the change in `/usr/local/demisto/upgrade_engine.sh` as follows: * If the `engine.connection.trust_any_certificate` configuration was set to true (trust any certificate), set the TRUST\_ANY\_CERT variable to -k. * If the `engine.connection.trust_any_certificate` configuration was set to false, the TRUST\_ANY\_CERT variable should be blank (““). 4. To check the connectivity from the engine to the main server, see [Troubleshoot Engine Connectivity](#UUID-f95c4d6a-95c0-53f5-291c-5598d004943a). 5. If the installation issue remains, open a support case with logs from the main server and engine. 1. On the engine server, in `/usr/local/demisto/d1.conf`, set "LogLevel": "debug”. 2. On the main server, navigate to **Settings** → **ABOUT** → **Troubleshooting** and verify that the Log Level is set to Debug. 3. Restart the d1 service and let it run for a few minutes. **`sudo systemctl restart d1`** 4. On the main server, go to **Settings** → **ABOUT** → **Troubleshooting** → **Download logs** to download a log bundle. 5. Capture a journalctl: **`journalctl --since "1 day ago" > engineTroubleshootingJournalctl.log`** 6. On the engine server, tar up the logs, conf, journalctl, and install log on the engine. **`tar -cvzf engineLogs.tar.gz /var/log/demisto /usr/local/demisto/d1.conf /tmp/demisto_install.log engineTroubleshootingJournalctl.log`** ### Troubleshoot Engine Upgrades During an upgrade, the upgrade file is sent to the engine server. A cron job running on the engine server checks if that file exists. The most common upgrade error is that the job is not running, so the new installer does not run. 1. SSH to the machine. 2. Check the d1 service status on the engine server. It is possible that it stopped or doesn't exist. **`sudo systemctl status d1`** 3. Access the installer log on the engine server and review the error. **`sudo vi /tmp/demisto_install.log`** 4. Rerun the installer on the engine server using one of the following options. You can open a second window and run **`watch df -h`**. If the problem seems to be disk space, you should resolve the disk space issue and then rerun the installer. 1. Option 1 1. Download the installer from the user interface and copy it to the engine server. 2. sudo chmod +x installer.sh 3. sudo ./installer.sh -- -y 2. Option 2 1. Verify that `/usr/local/demisto/d1_upgrade.sh` exists. If the file exists, run the following on the engine server: **`sudo chmod +x /usr/local/demisto/d1_upgrade.sh`** **`sudo /usr/local/demisto/d1_upgrade.sh`** 2. If **`d1_upgrade.sh`** does not exist, check if `/usr/local/demisto/archived_d1_upgrade.sh` exists and that it was created at the time of the attempted upgrade. If the file exists and was created at the time of the attempted upgrade, run the following on the engine server: **`sudo chmod +x /usr/local/demisto/archived_d1_upgrade.sh`** **`sudo /usr/local/demisto/archived_d1_upgrade.sh`** ### Troubleshoot Engine Connectivity The following provides instructions for troubleshooting connectivity issues from the engine to the endpoint. 1. Follow the instructions in [network troubleshooting](https://xsoar.pan.dev/docs/reference/articles/troubleshooting-guide#host-based-networking). 2. Ensure that the engine can reach the endpoint by running the following command on the server engine. **`sudo curl -kvv `** 3. If the engine could not reach the endpoint, try the IP with curl instruction adding the http(s)//, or try using ping. If this works, add the IP to the /etc/hosts file with the hostname and try to reach the endpoint again by running the following command on the engine server **`sudo curl -kvv .`** If this still fails, then this is an issue of connectivity between the engine and endpoint and you need to resolve this with your networking team. 4. Once connectivity has been confirmed via curl: * Try connecting within Docker without passing host networking. **`docker run -it --rm demisto/netutils:1.0.0.6138 curl -kvv `** If this succeeds but the integration still fails, it could be a integration credentials issue. In that case, open a [support case](https://support.paloaltonetworks.com/support). * If without passing host networking fails, run the following: **`docker run -it --rm --network=host demisto/netutils:1.0.0.6138 curl -kvv `** If this succeeds, add "python.pass.extra.keys": "--network=host" to /usr/local/demisto/d1.conf and retest the integration. If you see a Docker or Selinux issue, see [Troubleshoot Docker Networking Issues](/xsoar-6-administrator-guide/6.14/onboard-cortex-xsoar/docker/troubleshoot-docker-networking-issues.md). 5. If the installation issue remains, open a support case with logs from the main server and engine. 1. On the engine server, in `/usr/local/demisto/d1.conf`, set "LogLevel": "debug”. 2. On the main server, navigate to **Settings** → **ABOUT** → **Troubleshooting** and verify that the Log Level is set to Debug. 3. Restart the d1 service and let it run for a few minutes. **`sudo systemctl restart d1`** 4. On the main server, go to **Settings** → **ABOUT** → **Troubleshooting**+Download logsto download a log bundle. 5. Capture a journalctl: **`journalctl --since "1 day ago" > engineTroubleshootingJournalctl.log`** 6. On the engine server, tar up the logs, conf, journalctl, and install log on the engine. **`tar -cvzf engineLogs.tar.gz /var/log/demisto /usr/local/demisto/d1.conf /tmp/demisto_install.log engineTroubleshootingJournalctl.log`** #### Engine 443 Error This error might occur when a connection is established between an engine and the Cortex XSOAR server, because, by default, Linux does not allow processes to listen on low-level ports. #### Error Message **`listen tcp :443: bind: permission denied`** #### Solution * In the `d1.conf` file, change the port number to a higher one, for example, 8443. * Run this command: **`sudo setcap CAP_NET_BIND_SERVICE=+eip /path/to/binary`**. After running this command the server should be able to bind to low-numbered ports. ### Troubleshoot Integrations Running on Engines The following are common errors that occur when integrations are running on an engine: * [Troubleshoot Engine Import Error or Invalid Syntax Error](#UUID-11fb346f-bacd-d9c0-e72f-ac55863862c7) * [Troubleshoot Permission Denied](#UUID-672fa55d-4359-1e6d-b43c-cd90add2d887) #### Troubleshoot Engine Import Error or Invalid Syntax Error When running an integration on an engine, the most common errors are: * **`Broken Pipe`** * **`ImportError: No module named...`** * **`Invalid syntax`** * **`Script failed to run: exec: “python”: executable file not found in $PATH (2603)`** These errors could indicate that the engine is not using Docker. 1. Use SSH to access the engine server. 2. Make sure Docker is healthy. 1. Ensure that Docker is installed and is running. **`sudo systemctl status docker`** If the Docker status is not good, restart your Docker. **`sudo systemctl restart docker`** 2. Ensure Docker can run a container. **`sudo docker run hello-world`** If this fails, reinstall your Docker. 3. Access the d1.conf file on the engine server. **`sudo vi /usr/local/demisto/d1.conf`** 4. Add the **`"python.engine.docker": true`** configuration to the d1.conf file and remove any other configurations related to python and Docker, such as “python.executable.no.docker”. 5. Restart the system on the engine server. **`sudo systemctl restart d1`** 6. Retest the integration from the user interface. This may take a few minutes since it may need to pull the relevant Docker image. #### Troubleshoot Permission Denied A common error message you may see when running integrations on engines is: **`Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.35/images/json?t`**. 1. Determine if you are using a Docker group or Dockerroot group by running one of the following on the server engine: * **`ls -la /var/run/docker.sock`** The output from this command will show what user/group is running docker.sock. For example: ```programlisting srw-rw----. 1 root docker 0 Apr 12 20:32 /var/run/docker.sock ``` shows that it’s a Docker group and not Dockerroot. * **`cat /etc/group | grep docker`** This command shows if you are running Docker or Dockerroot.

Note

Docker CE installations typically run Docker, while Docker EE installations typically run Dockerroot.

2. To fix a Docker user, run the following commands on the server engine: 1. **`sudo groupadd docker`** 2. **`sudo usermod -aG docker demisto`** 3. **`sudo systemctl restart docker`** 4. **`sudo systemctl restart d1`** 3. To fix a Dockerroot user, run the following commands on the server engine: 1. **`sudo groupadd dockerroot`** 2. Set the dockerroot group in `/etc/docker/daemon.json`. For example: { "group": "dockerroot" }. 3. **`sudo usermod -aG dockerroot demisto`** 4. **`sudo chcon -Rt svirt_sandbox_file_t /var/lib/demisto/temp`** 5. **`sudo systemctl restart docker`** 6. **`sudo systemctl restart d1`** --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://cortex-docs.paloaltonetworks.com/xsoar-6-administrator-guide/6.14/onboard-cortex-xsoar/engines/troubleshoot-cortex-xsoar-engines.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.