Filter and transform data

Filter and transform data in Cortex XSOAR 8.10 On-prem playbooks.

In Cortex XSOAR, data is extracted and collected from various sources, such as playbook tasks, command results, and fetched incidents, and presented in JSON format. The data can be manipulated by using filters and transformers.

Filters

Filters enable you to extract relevant data which you can use elsewhere in Cortex XSOAR. For example, if an incident has several files with varying file types and extensions, you can filter the files by file extension or file type, and use the filtered files in a detonation playbook. You can filter as many objects as required. Cortex XSOAR automatically calculates the context root to which to filter. You can change the context root as necessary.

Caution

You can change the context data root to filter, but it is not recommended to select a different root, as it affects the filter results. The drop-down list displays the filter root for backward compatibility.

Transformers

Transformers modify or format data to make it suitable for further processing or presentation. For example, you can convert a date in non-Unix format to Unix format. Another example is applying the count transformer, which renders the number of elements. When you have more than one transformer, they apply in the order that they appear. You can reorder them using click-and-drag.

Add filters and transformers in a playbook task

1. Create or edit a playbook task.
2. In the field you want to add a filter or transformer (for example, inputs or outputs), click the curly brackets and then select Filters and Transformers.
3. In the Get field, type or select data you want to filter or transform. For example, EWS.Items.Name.
4. (Optional) To filter the data, do the following.

   1. In the Filter section, click Add filter.

      When adding a filter, the context root to filter is automatically populated.
   2. Select the data you want to filter.

      Note

      If you want to filter by a field name that contains a dot (.), you must escape it using a backslash (\ to prevent the system from interpreting it as a nested path. For example, to filter by the field @odata.type, use @odata\.type.
   3. Select the filter operators.
   4. Add the value.
   5. Click the checkbox to save the filter.
5. (Optional) To apply transformers to the field, click Add transformer.

   1. Click the transformer and select the relevant transformer.

      By default, the transformer is set to To upper case(String). Click it to pick a different transformer, for example to change the date format for when incidents occurred.
   2. Select the transformer operators.
   3. Click the tick box to save.
6. (Optional) To test the filter or transformation click Test and select the investigation or add it manually.

Example: Filter items with an EXE extension

In this example, we want to filter all EWS Item names that have the extension exe.

playbook-context.png

1. From the Filters & transformers window, in the Get field, type EWS.Items.Name to extract all Item names in EWS.

   The context root to filter is EWS,Items.

   

   filter-name.png
2. In the Filter section, click Add filter.
3. In the left-hand side, add Extension to the filter.
4. Select Equals (String) → ignore case.
5. In the right-hand side add exe.

   

   filter-exe.png
6. Click the tick box to save the filter.
7. Click Test.

   You should see Item names are filtered with the extension exe.

Example (advanced): Filter hostname for the last resolved time

In this example, we want to see the LastResolved time only from the demisto.com host name.

This is part of the data where we want to filter:

{
    "IP": [
       {
        "Address": "192.168.10.96",
        "AutoFocus": {
            "Resolutions": [
                {
                    "Hostname": "79463wwfqq,dattolocal.net",
                    "LastResolved": "2022-08-02 04:01:02"
                },
                {
                    "Hostname": "demisto.com",
                    "LastResolved": "2022-09-10 09:47:17"
                },
                {
                    "Hostname": "securesense.call4pchelp.com",
                    "LastResolved": "2022-04-22 11:49:06"
                }
            ]
        }
       },
       {
        "Address":"192.168.10.96",
        "AutoFocus": {
            "Resolutions":[
                {
                    "Hostname":"79463wwfqq,dattolocal.net",
                    "LastResolved":"2022-08-02 04:01:02"
                },
                {
                    "Hostname":"demisto.com",
                    "LastResolved":"2022-09-10 09:47:17"
                },
                {
                    "Hostname":"securesense.call4pchelp.com",
                    "LastResolved":"2022-04-22 11:49:06"
                }
            ]
        }
       }
    ]
}

1. From the Filters & transformers window, in the Get field, type IP.AutoFocus.Resolutions.LastResolve.

   

   playbook-filter-auto.png
2. In the Filter section, click Add filter.

   Cortex XSOAR automatically calculates that the context root to filter is IP.AutoFocus.Resolutions.

   

   playbook-filter-autores.png
3. In the left-hand side, add Hostname to the filter.
4. Select Equals (String) → Ends with
5. In the right-hand side add demisto.com.
6. Click the checkbox to save.

   

   playbook-filter-autofilter.png
7. Click Test.

   

   playbook-filter-autotest.png

Create custom filters and transformers

If you require a filter or transformer that is not provided out-of-the-box, you can create your own by creating a script and then adding to the operators window.

1. Select Incident Response → Automation → Scripts → New Automation.
2. Type a meaningful name for the script, and click Save.
3. To create a filter operator script, do the following:

   1. In the Tags field, add the filter tag.

      If you want a custom transformer that operates on an entire array rather than on each individual item, you need to add the entirelist tag.
   2. In the Arguments section, add the following arguments:

      Argument	Description
      left	Mark as mandatory. This argument defines the left-side value of the transformer operation. In this example, this is the value being checked if it falls within the range specified in the right-side value.
      right	Mark as mandatory. This argument defines the right-side value of the transformer operation. In this example, this is the range to check if the left-side value is in.

   3. Add the script syntax and save.
4. To create a transformer operator script do the following:

   1. In the Tags field, add the transformer tag.
   2. In the Arguments section, add the following arguments:

      Argument	Description
      value	Mark as mandatory. The value to transform. In this example, this is the UNIX epoch timestamp to convert to ISO format.

   3. Add the script syntax and save.
5. Go to the filters and transformers window and select the operator.

Filter considerations, categories, and built-in filters

You can use built-in filters to define your filter, they are grouped by category. Before defining a filter, consider the following.

Filter considerations

• Filters try to cast the transformed value and arguments to the appropriate type. The task fails if casting fails. For example, “a” Equals {“some”: “object”} => Error
• If the filter's left-side value expects a single item but receives a list, the filter passes if at least one item meets the requirements. For example, [“a”, “b”, “c”] Equals “b” => true.
• If the filter's left-side value expects a list but receives a single item, it converts it to a list with a single item. For example, “a” Contains “a” => True.
• Some custom filters are implemented as scripts with the filter tag. You can find examples in the playbook automation task description.
• Filters in conditional tasks do not iterate the items of the root. Instead, they fetch the left-side value and the right-side value and compare them.

Filter categories and built-in filters

When adding a filter, clicking the default Equals (String) field opens a search window showing the available built-in filters. They are defined by category as follows:

General

General filters such as Contains, Doesn’t Contain, In, and Is empty.

Filter	Description
Contains	Tests whether the value on the left is contained in the value on the right. Can be used for any kind of object (not limited to a string).
Doesn't Contain	Tests whether the value on the left is NOT contained in the value on the right. Can be used for any kind of object (not limited to a string).
Has length of	Tests whether a list specified on the left has the number of items specified on the right.
In	Tests whether the value on the left is contained in the object on the right.
Is defined	Tests whether a key on the left exists in context. Note Is defined considers false and empty strings and lists to be defined values. If you don't want those to be included as defined, use Is not empty.
Is empty	Tests whether the value of a key is empty.
Is not empty	Tests whether the value of a key is NOT empty.
Not defined	Tests whether a key on the left does NOT exist in context. Note Not defined considers false and empty strings and lists to be defined values. If you don't want those to be included as defined, use Is empty.
Not in	Tests whether the value on the left is NOT contained in the object on the right.

String

Determines the relationship between the left-side string value and the right-side string value, such as starts with, includes, and in the list. The string filter returns partial matches as True.

Filter	Description
Doesn't end with	Tests whether the string on the left is NOT the end of the string on the right.
Doesn't equal	Tests whether the strings are NOT the same.
Doesn't include	Tests whether the string on the right is NOT a substring of the string on the left.
Doesn't start with	Tests whether the string on the right is NOT the beginning of the string on the left.
Ends with	Tests whether the string on the left is the end of the string on the right.
Equals	Tests whether the strings are the same.
Has length	Tests whether the two strings have the same length.
In list	Tests whether the string on the left is in the list on the right.
Includes	Tests whether the string on the right is a substring of the string on the left.
Matches - regex	Tests whether the string on the left matches the regex on the right. Uses Go-style regex.
Not in list	Tests whether the string on the left is NOT a substring of the string on the right.
Starts with	Tests whether the string on the right is the beginning of the string on the left.
StringContainsArray	Tests whether a substring or an array of substrings on the left is within a string array on the right. Supports single strings as well. For example, for substrings ['a', 'b', 'c'] in string 'a' the script returns true.

Number

Determines the relationship between the left-side number value and the right-side number value, such as Equals, Greater than, and Less than.

Filter	Description
Doesn't equal	Tests whether the number on the left does NOT equal the number on the right.
Equals	Tests whether the number on the left equals the number on the right.
Greater or equal	Tests whether the number on the left is greater than or equal to the number on the right.
Greater than	Tests whether the number on the left is greater than the number on the right.
InRange	Tests whether the number on the left is within a range specified on the right. For example, if the left value is 4, and the range on the right is 1,8, the condition is true.
Less or equal	Tests whether the number on the left is less than or equal to the number on the right.
Less than	Tests whether the number on the left is less than the number on the right.

Date

Determines whether the left-side time value is earlier than, later than, or the same time as the right-side time value.

Filter	Description
After	Tests whether the date on the left is after the date on the right.
AfterRelativeDate	Tests whether the date on the left occurred after the provided relative time (such as '6 months ago') on the right. Returns True or False.
Before	Tests whether the date on the left is before the date on the right.
Same as	Tests whether the two dates are the same.

Boolean

Determines whether a field is true or false, or the string representation is true or false.

Filter	Description
Is false	Tests whether the value on the left evaluates to false.
Is true	Tests whether the value on the left evaluates to true.

Other

Miscellaneous filters, including CheckIfSubdomain and IsInCidrRanges.

Filter	Description
CheckIfSubdomain	Tests whether the value on the left is a subdomain of the value on the right.
CIDRBiggerThanPrefix	Tests whether the CIDR prefix on the left is bigger than the defined maximum prefix on the right.
GreaterCidrNumAddresses	Tests whether the number of available addresses in IPv4 or IPv6 CIDR on the right is greater than the input given on the left.
IsInCidrRanges	Tests whether the IPv4 address on the left is contained in at least one of the comma-delimited CIDR ranges on the right. Multiple IPv4 addresses can be passed in a comma-delimited list and each address is tested.
IsNotInCidrRanges	Tests whether the IPv4 address on the left is NOT contained in at least one of the comma-delimited CIDR ranges on the right. Multiple IPv4 addresses can be passed in a comma-delimited list and each address is tested.
IsRFC1918Address	Tests whether an IPv4 address on the left is in the private RFC-1918 address space (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) on the right.
LowerCidrNumAddresses	Tests whether the number of available addresses in IPv4 or IPv6 CIDR on the right is less than the input given on the left.

Transformer considerations, categories, and built-in transformers

You can use built-in transformers to define your transformer, they are grouped by category. Before defining a transformer, consider the following.

Transformer considerations

• Transformers try to cast the transformed value (and arguments) to the necessary type. Tasks will fail if casting has failed, for example {“some”: “object”} To upper case => Error.
• Some transformers are applied on each item of the result. For example, a, b, c To upper case => A, B, C.
• Some transformers operate on the entire list. For example, a, b, c count => 3.
• Some custom transformers are implemented as scripts with the transformer tag. You can find examples in the playbook automation task description.

Transformer categories and built-in transformers

When adding a transformer, clicking the default To upper case (String) field opens a search window showing the available built-in transformers. They are defined by category as follows.

Transformer category	Description	Built-in transformers
General	Generic transformers	FIXME_ACCORDION_PLACEHOLDER
String	String transformers Note To make regex case non-sensitive, use the (?i) prefix (for example (?i)yourRegexText.	FIXME_ACCORDION_PLACEHOLDER
Number	Number transformers	FIXME_ACCORDION_PLACEHOLDER
Date	Date transformers	FIXME_ACCORDION_PLACEHOLDERFIXME_ACCORDION_PLACEHOLDER