secrets

Detect and manage secrets in Cortex content with the Demisto SDK.

Run the secrets validator to catch sensitive data before exposing code to a public repository.

Attach the full path to manually specify an allow list.

Note

This command is not guaranteed to find all secrets. A manual review of all files is highly recommended.

Arguments

Argument	Description
-i, --input	Specifies a file to check secret for.
--post-commit	Whether the secrets validation is done after committing files. This helps the command determine which files it should check in its run. Before you commit the files it should not be used. Mostly for build validations. The default value is False.
-ie, --ignore-entropy	Ignores the entropy algorithm that finds secret strings (passwords/API keys).
-wl <path to allow list file>, --whitelist <path to allow list file>	Full path to the allow list file. The file name should be secrets_white_list.json. The default is ./Tests/secrets_white_list.json
--prev-ver	The branch against which to run secrets validation.

Examples

• demisto-sdk secrets

  Runs the secrets validator on uncommitted files.
• demisto-sdk secrets -i ./Packs/FeedAzure/Integrations/FeedAzure/FeedAzure.yml

  Runs the secrets validator on the file located in ./Packs/FeedAzure/Integrations/FeedAzure/FeedAzure.yml.
• demisto-sdk secrets --post-commit

  Runs the secrets validator on files after they are committed.
• demisto-sdk secrets -wl ./secrets_white_list.json

  Runs the secrets validator on files using a custom allow list file located in ./secrets_white_list.json.

More about secrets and sensitive data

The content repository is public and open source. It is important to not commit secrets and sensitive data into this repository.

Secret detection is done in the pre-commit stage and also in the build stage.

Caution

If actual secrets are detected in the build stage, it means they were already exposed on a public repository and the relevant people should be alerted.

Be careful to not post sensitive data in PR comments and code review.

Data considered sensitive:

• Customer identifying data, anything that can identify an organization as a customer
• Customer environment information
• Passwords
• IPs
• URLs/domains
• Email addresses not for testing
• PII (personal identifying information)
• Screenshots of third party products
• Screenshots that may contain any of the above data

Allow lists

• Temporary allow lists occur automatically for .yml files with context paths configured.
• The main allow list file is secrets_white_list.json. This file is divided into three main parts: IOCs, Generic Strings, and Integration specific strings.
• IOCs are divided further into types such as IPv4, IPv6, emails, and URLs.
• Secrets found via regex are only tested against the IOCs allow list. If an indicator is added to an allow list, it must be added to the IOCs dictionary.
• Generic strings are common words that appear in integrations and scripts.
• If an integration requires a specific key word it can be added to the dictionary with a key that is similar to the relevant integration. This is only relevant for the uncommon situation that the string does not fit logically anywhere else.

  For example, a lot of false positives with the term CookieMonster are found. The term CookieMonster is not relevant to the generic words, so it is relevant to create a new key in the file named sesame street with the value set to a list containing cookiemonster.
• Once the allow list file is updated with a string, it is globally allowed for all integrations, even if it is integration-specific.
• Only words with five or more characters are considered in the allow list.
• Secrets found in content packs are checked against both the allow list file provided in the -wl or --whitelist argument and in and the pack secrets file (.secrets-ignore).
• All words in an allow list must be lowercase. To make strings lowercase, use command+shift+u.

Ignore single or multi lines

Ignore lines instead of allow listing them when:

• The lines have dynamic secret/false positives (like a hash).
• There are very long, random strings (like a regex).
• It does not make sense for the lines to be on a shared allow list.

Phrases do not have to be in a comment, and not on their own, so you can mix them with any line.

Example for ignoring a single line in Python:

i_wrote_too_many_words_without_any_separator =  ReadableContentsFormat.SomeExample # disable-secrets-detection

Example for ignoring multi lines in Python:

# About to drop some mean regex right now disable-secrets-detection-start
TONS_OF_REGEX = r'(?:(?:[0-9A-Fa-f]{1,4}:){6}(?:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}|(?:(?:[0-9]|[1-9][0-9]|1' \
             r'[0-9]{2}|2[0-4][0-9]|25[0-5])\\.){3}(?:[0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]))|::'\
             r'(?:[0-9A-Fa-f]{1,4}:){5}(?:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}|(?:(?:[0-9]|[1-9][0-9]|1[0-9]'\
             r'{2}|2[0-4][0-9]|25[0-5])\\.){3}(?:[0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]))|'\
             r'(?:[0-9A-Fa-f]{1,4})?::(?:[0-9A-Fa-f]{1,4}:){4}(?:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4}|'\
             r'(?:(?:[0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\\.){3}(?:[0-9]|[1-9][0-9]|1[0-9]{2}'\
             r'|2[0-4][0-9]|25[0-5]))|(?:[0-9A-Fa-f]{1,4}:[0-9A-Fa-f]{1,4})?::(?:[0-9A-Fa-f]{1,4}:){3}?)'
# Drop the mic disable-secrets-detection-end

How secrets detection works

• The more random and longer a string is, the higher its entropy. The average string has entropy of between 2 to 3.4, but some longer strings of variable names can have entropy of 3.6 and greater.
• Regex is used to detect mostly IOCs with typically low entropy. For example, IPv4 can have a lot of repeating characters, reducing its randomness.
• Secrets detection is done on the file level, on the line level, and then on the string level.
• Files that are currently scanned are added/modified files of formats: .yml, .py, .json, .md, .txt, .sh, .ini, .eml, .csv, .js, .pdf, and .html.
• eml and playbook files are only tested for indicators using regex.
• Long base64 strings are automatically ignored.
• If PDF file parsing fails, a warning is issued under the commit message and the file is skipped.

How allow lists work

• If a Python file is detected, a related YAML file is automatically pulled and its context paths are used in a temporary allow list.
• Currently, regex is used to identify strings that are likely to be high entropy strings and are regarded as false positives, for example, dates and UUID. Regex is also used to catch and remove patterns that have high probability of being false positives.