URL extraction

Cortex XSIAM guidance for extracting URL indicators from text.

The Cortex XSIAM URL indicator type is built using regular expression and a formatting script. The following describes the URL extraction components and what output you should expect when extracting URL indicators.

Cortex XSIAM URL indicator extraction components

There are two components when extracting URL indicators:

• Regular expression
• Formatting script

URL indicator regular expression

From a given text, a URL regular expression tries to catch a valid URL based on the following characteristics:

• A URL prefixed by one of the following protocols:

  • HTTP
  • HTTPS
  • FTP
  • FTPS
  • HXXP (defanged HTTP)
  • HXXPS (defanged HTTPS)
• A URL with ASCII or non-ASCII characters
• Escaped and unescaped URLs
• URL with or without query parameters

Format extracted URL indicators

After extracting the URL using regular expression, a FormatURL formatting script iterates on each given URL and does the following:

1. If the URL is prefixed by a URL defense system, Proofpoint or ATP, the script extracts the redirected URL and continues with steps 3-6 for the original and extracted redirected URL.
2. If the URL is NOT prefixed by a URL defense system, Proofpoint or ATP, the script checks if the first query parameter is a redirected URL query parameter by checking if the first parameter value starts with HTTP or HTTPS.

   For example:

   https://www.good.site/index.html?redirectURL=https://evil.com/mal.html

   If the query parameter exists, the script extracts the redirected URL and performs steps 3-6 both for the given URL and the one extracted from the query parameter.
3. Replaces "[.]" with "." .

   For example:

   https://www[.]example.com becomes https://www.example.com
4. Decodes the URL.

   For example:

   https://www.example.com%2F%21%40 becomes https://www.example.com/!@
5. Converts obfuscated characters.

   For example:

   hxxp → http becomes hxxps → https
6. Returns the formatted URL.

Supported URL indicator structures

The following are the most common supported URL structures:

• http://öevil.tld/
• https://evilö.tld/evil.html
• www.evilö.tld/evil.aspx
• https://www.evöl.tld/
• www.evil.tld/resource
• http://xn--e1v2i3l4.tld/evilagain.aspx
• https://www.xn--e1v2i3l4.tld
• hxxps://www.xn--e1v2i3l4.tld
• hxxp://www.xn--e1v2i3l4.tld
• www.evil.tld:443/path/to/resource.html
• https://1.2.3.4/path/to/resource.html
• 1.2.3.4/path
• 1.2.3.4/path/to/resource.html
• http://1.2.3.4:8080/
• http://1.2.3.4:8080/resource.html
• http://☺.evil.tld/
• http://1.2.3.4
• ftp://foo.bar/resource
• ftps://foo.bar/resource

For more information, see Indicator extraction.