> For the complete documentation index, see [llms.txt](https://cortex-docs.paloaltonetworks.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cortex-docs.paloaltonetworks.com/cortex-xsiam-developer-guide/cortex-xsiam-development-guide/indicators/url-extraction.md).

# URL extraction

The Cortex XSIAM URL indicator type is built using regular expression and a formatting script. The following describes the URL extraction components and what output you should expect when extracting URL indicators.

#### URL extraction components

There are two components when extracting URL indicators:

* Regular expression
* Formatting script

**Regular expression**

From a given text, a URL regular expression tries to catch a valid URL based on the following characteristics:

* A URL prefixed by one of the following protocols:
  * HTTP
  * HTTPS
  * FTP
  * FTPS
  * HXXP (defanged HTTP)
  * HXXPS (defanged HTTPS)
* A URL with ASCII or non-ASCII characters
* Escaped and unescaped URLs
* URL with or without query parameters

**Formatting script**

After extracting the URL using regular expression, a `FormatURL` formatting script iterates on each given URL and does the following:

1. If the URL is prefixed by a URL defense system, Proofpoint or ATP, the script extracts the redirected URL and continues with steps 3-6 for the original and extracted redirected URL.
2. If the URL is NOT prefixed by a URL defense system, Proofpoint or ATP, the script checks if the first query parameter is a redirected URL query parameter by checking if the first parameter value starts with HTTP or HTTPS.

   For example:

   `https://www.good.site/index.html?redirectURL=https://evil.com/mal.html`

   If the query parameter exists, the script extracts the redirected URL and performs steps 3-6 both for the given URL and the one extracted from the query parameter.
3. Replaces "\[.]" with "." .

   For example:

   `https://www[.]example.com` becomes `https://www.example.com`
4. Decodes the URL.

   For example:

   `https://www.example.com%2F%21%40` becomes `https://www.example.com/!@`
5. Converts obfuscated characters.

   For example:

   `hxxp → http` becomes `hxxps → https`
6. Returns the formatted URL.

#### Common URL structures

The following are the most common supported URL structures:

* <http://öevil.tld/>
* <https://evilö.tld/evil.html>
* [www.evilö.tld/evil.aspx](http://www.evilö.tld/evil.aspx)
* <https://www.evöl.tld/>
* [www.evil.tld/resource](http://www.evil.tld/resource)
* [http://xn--e1v2i3l4.tld/evilagain.aspx](https://cortex-docs.paloaltonetworks.com/cortex-xsiam-developer-guide/cortex-xsiam-development-guide/indicators/http:/xn--e1v2i3l4.tld/evilagain.aspx)
* [https://www.xn--e1v2i3l4.tld](https://cortex-docs.paloaltonetworks.com/cortex-xsiam-developer-guide/cortex-xsiam-development-guide/indicators/https:/www.xn--e1v2i3l4.tld)
* hxxps\://[www.xn--e1v2i3l4.tld](https://cortex-docs.paloaltonetworks.com/cortex-xsiam-developer-guide/cortex-xsiam-development-guide/indicators/http:/www.xn--e1v2i3l4.tld)
* hxxp\://[www.xn--e1v2i3l4.tld](https://cortex-docs.paloaltonetworks.com/cortex-xsiam-developer-guide/cortex-xsiam-development-guide/indicators/http:/www.xn--e1v2i3l4.tld)
* [www.evil.tld:443/path/to/resource.html](http://www.evil.tld:443/path/to/resource.html)
* <https://1.2.3.4/path/to/resource.html>
* 1.2.3.4/path
* 1.2.3.4/path/to/resource.html
* <http://1.2.3.4:8080/>
* <http://1.2.3.4:8080/resource.html>
* http\://☺.evil.tld/
* <http://1.2.3.4>
* ftp\://foo.bar/resource
* ftps\://foo.bar/resource

For more information, see [Indicator extraction](/cortex-xsiam-developer-guide/cortex-xsiam-development-guide/indicators.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://cortex-docs.paloaltonetworks.com/cortex-xsiam-developer-guide/cortex-xsiam-development-guide/indicators/url-extraction.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.