Paths and URLs

Include paths/URLs and Avoid paths/URLs are under Scan preferences > What should we scan? in the scan profile settings. Both take root-relative paths, such as /blog.

Include Paths/URLs

Paths in this list are added as starting points for crawling. Use it for pages the crawler can’t reach by following links on your site.

Including a path doesn’t limit the scan to that path, and it doesn’t guarantee the path gets tested. Like everything the crawler finds, an included path is evaluated for whether it’s relevant for testing. Duplicates of a page already seen, pages that can’t be interacted with, and client-only URLs are examples of URLs that are left out. The limits of a scan also still apply, so if the scope of the scan is too large, the scan may not reach an included path.

Avoid Paths/URLs

The scanner doesn’t send requests to paths in this list, during crawling or during testing. Use it for pages the scanner shouldn’t touch, such as your logout link or a page that deletes data, and for parts of the site that take a lot of scan time, such as long lists of blog posts or product pages.

For example, avoiding /blog on a scan profile for example.com means the scan won’t make requests to any URL starting with http(s)://example.com/blog.

Avoided paths also block the resources your pages load from them, such as JavaScript files. If you avoid a path your site loads scripts from, pages can render blank and the scan misses everything on them. Avoiding the root path / blocks every resource under it, unless you add the resource to Include paths/URLs.

An avoided path applies to the scan profile’s domain and to every subdomain the scan covers. You can’t avoid a path on one subdomain only, such as shop-prod.example.com/products. Either avoid /products everywhere, or add shop-prod to Avoid subdomains.

Scan Only Part of a Site

When a path is in both lists, the include wins. Use the two lists together to test one part of an application:

To scan only the blog on example.com, avoid / and include /blog/. The scan crawls and tests only URLs starting with http(s)://example.com/blog/, and doesn’t request the root page.
If you have many product pages with the same structure under /items/, avoid /items/ and include /items/1001. The scan tests one product page instead of spending time mapping all of them.

Including /products/ covers the paths under it. If a deeper path such as /products/notebooks/ is still missing from the Crawled URLs finding after a new scan, add it to Include paths/URLs as well.

Wildcards

Both lists accept * as a wildcard. For example, /product/*/details matches /product/5/details and /product/6/details. You can use any number of wildcards in a path, such as /blog/*/guestblog/*/details.

A trailing wildcard is implied: /admin works the same as /admin*. Once a path contains an asterisk, the trailing wildcard isn’t implied, so /product/*/details is not the same as /product/*/details*.

Regular expressions aren’t supported.

The path lists are passed to the scanner at the start of the scan, so avoided paths are also respected while Recorded Login and Recorded Crawling replay.

Check What Was Crawled

The Crawled URLs finding in the scan results lists the URLs the scan crawled. Use it to confirm that included paths were reached and avoided paths were left alone.

Next Steps

Subdomains — Control which subdomains the scan covers
Recorded Crawling — Cover flows the crawler can’t complete on its own
Settings — All scan profile settings