Podman

Use Podman engines with Cortex XSOAR On-prem 8.7 (EoL).

Podman is a daemonless container engine for developing, managing, and running OCI Containers on the Linux System. Containers can either be run as root or in rootless mode.

If you use the Shell installer to install an engine, Cortex XSOAR automatically detects the container management type based on the operating system. For example, if your operating system is running RHEL v8 and higher, Cortex XSOAR installs Podman packages and configures the operating system to enable Podman in rootless mode.

Note

When upgrading an engine, the engine keeps the previously used container management type (regardless of distribution version).

If using PowerShell integrations, you may need to configure the default SELinux policy as Podman can affect processes which mmap to /dev/zero.

Docker hardening guidelines

Docker hardening guidelines can be applied to Podman, with the exception of Limit Available Memory, Limit Available CPU, and Limit PIDS.

Change container storage directory

By default, Podman uses the $HOME/.local/share/containers/storage directory. To use a different directory for container storage, edit the Podman config file located at /home/demisto/.config/containers/storage.conf. If the Podman config file does not exist, you need to create it and change the ownership.

The new storage directory needs to be owned by the demisto user, otherwise they will be denied access to it.

Warning

Do not use NAS storage or a temporary (tmpfs) directory for the graphroot setting. The graphroot needs to be a local, non-temporary directory for Podman to work. For more information, see https://en.wikipedia.org/wiki/Network-attached_storage.

Tip

We recommend reserving 150 GB for container storage, either in the /home partition or a different storage directory that you have set using the grarphroot key.

1. If the Podman config file does not exist:

   1. Create the Podman config file.

      sudo mkdir -p /home/demisto/.config/containers

      cp /etc/containers/storage.conf /home/demisto/.config/containers
   2. Change the ownership of the Podman config file.

      sudo chown -R demisto:demisto /home/demisto
2. To set a different directory for container storage, change the key: graphroot in the storage.conf file. For example:

   graphroot = "/var/lib/containers/xsoar-storage"
3. Some additional changes are required in the storage.conf file. Comment out the runroot setting by adding a # (hash) before it. For example:

   #runroot = "/run/containers/storage"

   Note

   Alternatively, the runroot setting may be set to some temporary directory that is accessible by user demisto. If you choose to set the runroot, it must be a directory that is mounted as tmpfs (temporary filesystem), unlike the graphroot.
4. Under [storage.options.overlay], uncomment the following line (remove the # from the start):

   mount_program = "/usr/bin/fuse-overlayfs"
5. If the engine has already been installed, apply your changes to any existing containers:

   sudo -u demisto podman system migrate
6. Verify the change (once the engine is installed):

   sudo -u demisto podman info | grep graph

Install Podman

When installing a new engine on RHEL 8 or later, the shell installer configures Podman automatically. There are some cases, however, where you might need to install Podman manually:

• When using an installation method other than the shell installer (e.g. an RPM package) on RHEL 8 or later.
• When the shell installer did not successfully install Podman.
• When you want to migrate from Docker to Podman, for an existing Cortex XSOAR engine.

Note

• This procedure is intended for RHEL 8 or later. It may not work for other operating system types.
• Do not use NAS storage for the $HOME directory. The directory needs to be a local directory for Podman to work.

1. For RHEL 8, install Podman by typing the following commands:

   • sudo yum -y install slirp4netns fuse-overlayfs
   • sudo yum -y module install container-tools

   For RHEL 9 or later, install Podman by typing the following command:

   • sudo yum -y install slirp4netns fuse-overlayfs podman
2. Run the following commands:

   • sudo touch /etc/subuid /etc/subgid
   • sudo mkdir -p /home/demisto
   • sudo chown demisto:demisto /home/demisto
3. Configure the unqualified-search-registries used by Podman.

   Podman by default uses the fedoraproject.org, redhat.com, and docker.io unqualified search registries. SinceCortex XSOAR images use only the docker.io registry, you can speed up download times for container images by setting unqualified-search-registries to just docker.io.

   1. Create or edit the /home/demisto/.config/containers/registries.conf config file.
   2. In the file, set unqualified-search-registries = ["docker.io"]

      Note

      If you edit the file with the root user, make sure to set the demisto user as file owner by running chown demisto:demisto /home/demisto/.config/containers/registries.conf
4. Change the subuids and subgids by running the following command:

   sudo usermod --add-subuids 200000-265535 --add-subgids 200000-265535 demisto
5. Migrate existing containers to Podman:

   sudo sh -c "cd /; runuser -u demisto -- podman system migrate"
6. Set the net.ipv4.ping-group-range, by typing the following commands:

   • sudo sh -c "echo 'net.ipv4.ping_group_range=0 2000000' > /etc/sysctl.d/demisto-ping.conf"
   • sudo sysctl -w "net.ipv4.ping_group_range=0 2000000"
7. As root user, edit the following config file:

   /usr/local/demisto/d1.conf
8. Change the "container.engine.type": "docker" to “podman".

   If this line does not exist, add the following line to the file:

   "container.engine.type": "podman"

   "Server": {
                   "HttpsPort": "443",
                   "ProxyMode": true
           },
           "container": {
                                   "engine": {
                                           "type": "podman"
                                   }
           },
           "db": {
                   "index": {
                           "entry": {
                                   "disable": true

9. If the engine is running, restart the service.

   sudo systemctl restart d1

   Note

   If the Allow running multiple engines on the same machine option is selected, run the command:

   sudo systemctl restart d1_<Engine _name>

Migrate From Docker to Podman

Although Podman is set up automatically in an engine installation, it is possible to migrate from Docker to Podman in an existing engine. Follow the Podman installation instructions to migrate.

Troubleshoot Podman

dbus-daemon process leak

Podman version 3.4.1 and lower has a known issue that dbus-daemon processes may leak when running in an environment containing the dbus-x11 OS package. The issue occurs when the dbus-x11 OS package is installed, for example when installing an X11 desktop environment like GNOME desktop on the host machine. If you experience this issue, you see a large number of dbus-daemon processes owned by the demisto OS user. To check if you are affected by the issue, run the following command:

ps -fe | grep demisto | grep dbus-daemon

To fix this issue:

1. Remove the dbus-x11 OS package and dependent packages by running the following command:

   sudo yum remove dbus-x11
2. After removal you can kill the leaked dbus-daemon processes by running the following OS command:

   pgrep -u demisto dbus-daemon | xargs sudo kill

Invalid argument error

When Podman fails to run with an “Invalid argument” error, such as:

ERRO[0000] running `/usr/bin/newuidmap 15936 0 1029 1 1 165536 65536 65537 200000 65536`: newuidmap: write to uid_map failed: Invalid argument
Error: cannot set up namespace using "/usr/bin/newuidmap": exit status 1

This can be caused by duplicate lines for Cortex XSOAR in /etc/subuid and /etc/subgid.

To fix this issue:

1. Check if the /etc/subuid file contains multiple lines that start with the Cortex XSOAR username (usually demisto). For example:

   alice:100000:65536
   demisto:165536:65536
   demisto:200000:65536
   splunk:331072:65536

2. If this is the case, edit the file as root, and remove the extra line(s) for Cortex XSOAR. The line you should keep is the one that ends with 200000:65536. Continuing with the above example, here is the end result:

   alice:100000:65536
   demisto:200000:65536
   splunk:331072:65536

3. Repeat the above steps for the /etc/subgid file.

Verify Podman installation

When encountering errors in Cortex XSOAR that are Podman related, such as:

• failed to run "docker ps". stderr: [], err: [Timeout. Process killed (1400)
• Timeout while waiting for pong response [error 'Read timed out (15s)
• Error: error joining network namespace of container 06b8aec6eabe2e735128e3a72cb06c8ae2d97ade60a56ab555034442ea4e2a84: error retrieving network namespace at /tmp/podman-run-989/netns/cni-86dca01c-bd84-1aaf-85fb-72b659a8e42a: unknown FS magic on "/tmp/podman-run-993/netns/cni-86dca01c-bd84-1aaf-85fb-72b659a8e42a": 58465342

1. Verify that Podman is running properly with the demisto OS user, by performing the following steps:

   • Change the OS user to demisto by running the following command:

     sudo su - -s /bin/bash demisto
   • Check that your system complies with the minimum requirements, and view general system information such as host architecture, CPU, OS, registries, container storage path, etc., by running the following command:

     podman info
   • Check all active running containers, container names and IDs, by running the following command:

     podman ps
   • Check that Podman is able to run a container, by running the following command:

     podman run --rm -t demisto/python3:3.10.4.29342 echo "podman is working"

   If any of the Podman commands are not working, try running with the --log-level=debug to receive additional details as to why it is failing. For example:

   podman --log-level=debug ps

   podman --log-level=debug ps podman --log-level=debug run --rm -t demisto/python3:3.10.4.29342 echo "podman is working"
2. Reset the Podman Data Directories.

   If the Podman commands in step 1 are failing, you should clean the Podman working directories. Sometimes Podman's data directories get corrupted (for example, as a result of insufficient disk space).

   Note

   This step removes all Podman images including any custom images you may have created.

   1. Stop the engine by running the following command:

      sudo systemctl stop d1
   2. Ensure that all Podman containers of the demisto user are stopped, by running the following command:

      ps -fe | grep demisto | grep 'podman run'

      If required, kill the running containers.
   3. Delete the following directories (assuming the demisto OS user's home directory is at: /home/demisto)

      • sudo rm -rf /home/demisto/.cache/containers/
      • sudo rm -rf /home/demisto/.local/share/containers/
      • sudo rm -rf /tmp/podman-run-$(id -u demisto)
      • sudo rm -rf /tmp/containers-user-$(id -u demisto)
      • sudo rm -rf /tmp/tmp/run-$(id -u demisto)

      Note

      $(id -u demisto) is used to get the demisto user ID, which is part of the directory name. For example, /tmp/podman-run-993

      Not all the directories above may be present.
   4. Start the engine by running the following command:

      sudo systemctl start d1
   5. Verify that Podman is working properly with the demisto OS user by following step 1.

Unused containers taking up resources

In some cases, if the Podman process crashes or is killed abruptly it can leave containers on disk. You might see errors such as error allocating lock for new container: allocation failed; exceeded num_lock when the maximum number of locks used to manage containers is exhausted due to the unused containers that remain.

1. Change to the demisto operating system user sudo su - -s /bin/bash demisto.
2. Run podman ps -a -f status=exited to check for unused containers.
3. Clean up the unused containers podman container cleanup --rm -a.

   Note

   When you run podman container cleanup --rm -a, you might see a message such as running or paused containers cannot be moved without force. The message can be safely ignored, as it only pertains to current running containers, which are not removed.
4. After cleanup, verify there are no remaining unused containers podman ps -a -f status=exited.

Keyring quota exceeded error

Script failed to run: Docker code runner got container error: [Docker code script is in inconsistent state, ... error: [exit status 126] stderr: [Error: OCI runtime error: crun: create keyring ...: Disk quota exceeded]

By default, Podman creates a keyring that is used by each container. The limit per user on the machine might be low and Podman can reach the limit when running more containers than the keyring limit. To check the keyring usage, run the sudo cat /proc/key-users operating system command.

The command returns the usage for each UID (to retrieve the demisto user UID, run id demisto ). The fourth column shows the number of keys used out of the total number available. For more information about keys, see Kernel Key Retention Service.

You can either increase the limit of max keyrings (increasing to 1000 is safe and reasonable) per user as specified by your Linux vendor documentation or you can disable keyring creation by Podman. We recommend disabling keyring creation unless keyrings are used by Podman in other applications on the machine. To disable keyring creation by Podman, modify the containers.conf file and add the option keyring = false under the "[containers]" section. For more information, see the Containers Engine Configuration File.

error "exit status 125" and output "Error: chown ... operation not permitted "

If the container storage directory is not owned AND exclusively used by the demisto user, scripts will fail to run. See the Podman section for more information about assigning ownership of the storage directory.

Report a support case for installation issues

If the procedure set out in the Verify Podman installation section above does not solve the Podman issue and you require assistance from Support, do the following:

1. Include the following files as part of the support case:

   • /etc/containers/storage.conf
   • /home/demisto/.config/containers/storage.conf

     If the file does not exist, indicate that there is no such file.
   • /home/demisto/.config/containers/registries.conf

     If the file does not exist, indicate that there is no such file.
2. Include the output of the following commands as the demisto user.

   Note

   To change to the demisto OS user, run the following command:

   sudo su - -s /bin/bash demisto

   • podman info
   • podman images
   • podman --log-level=debug ps
   • podman --log-level=debug run --rm -t demisto/python3:3.10.4.29342 echo "podman is working"

Permission issues with directories under /run path

When installing a Cortex XSOAR engine on a RHEL system (version 8 or later), or when running an integration on such an engine, you get a permission error for a path under /run (for example /run/user/0 or /run/libpod).

1. In RHEL 9 only: Make sure the container-tools meta-package is installed, by running:

   yum -y install container-tools
2. Run the following commands:

   • cp /etc/containers/storage.conf /home/demisto/.config/containers/storage.conf
   • chown demisto:demisto /home/demisto/.config/containers/storage.conf
   • chmod 600 /home/demisto/.config/containers/storage.conf
3. Edit /home/demisto/.config/containers/storage.conf.

   • Under [storage], change runroot to some temporary directory that is accessible by user demisto.

     For example: runroot = "/tmp/podman-run-xsoar"

     Important

     The runroot must be located under the tmpfs file system type. This is required to clean Podman's run state on reboot and for performance reasons.
   • Also under [storage], change graphroot (where container images are stored) to any location that is owned and accessible by user demisto. We recommend using this standard path:

     graphroot = "/home/demisto/.local/share/containers/storage"

     Caution

     Unlike the runroot, the graphroot must NOT be located under the tmpfs file system type. Using tmpfs for the graphroot might corrupt container images, causing command executions to fail. It also degrades performance by forcing Podman to needlessly re-pull images.
   • Under [storage.options.overlay], uncomment the following line (remove the # from the start):

     mount_program = "/usr/bin/fuse-overlayfs"
4. Save the file and run the following.

   Note

   You must switch to user demisto before running the "system migrate" (running it as root will have no effect).

   • su - demisto
   • podman system migrate
5. Also as user demisto, run the following to ensure the path changes were applied:

   podman info | grep Root

   You should see the correct runRoot and graphRoot settings.
6. As user demisto, verify the issue is resolved by running:

   podman run hello-world
7. If the issue persists, purge Podman's database by running the following:

   Note

   The "system migrate" must be done by user demisto.

   • rm -rf /home/demisto/.local/share/containers/*
   • podman system migrate