6 min read
Also: XML sitemap, Sitemaps protocol, sitemap index, sitemap.xml.gz, sitemaps.org, sitemap.xml example
sitemap.xml
Quick answer
sitemap.xml is an XML file listing a site's URLs, each in a <url><loc> entry with an optional <lastmod> date, so search engines can discover them; one file holds up to 50,000 URLs or 50 MB uncompressed. Submit it in Google Search Console or add a Sitemap: line to your robots.txt.
sitemap.xml is an XML file listing a site's URLs, and optionally when each last changed, so search engines can find them. It follows the Sitemaps 0.9 protocol at sitemaps.org, with a limit of 50,000 URLs or 50 MB uncompressed per file.
Last verified
What people do with sitemap.xml
- Add a Sitemap line to robots.txt
robots.txt syntax, examples and common mistakes.
- Use a sitemap to plan a crawl
How crawlers discover pages from sitemaps and links.
- Watch a sitemap for new pages
The "A web page changes" trigger fetches a URL and fires on change.
- Get alerts when a website changes
Set up change alerts for pages or sitemaps.
- Fetch a sitemap on a schedule with an HTTP step
Download the XML and pass it to later steps.
Key facts
| Protocol version | Sitemaps 0.9 (sitemaps.org) |
|---|---|
| Backed by | Google, Yahoo and Microsoft jointly since 2006 |
| Max URLs per file | 50,000 |
| Max file size | 50 MB uncompressed |
| Tags Google uses | loc and lastmod (if accurate); priority and changefreq are ignored |
| How crawlers find it | Sitemap: line in robots.txt, or Search Console / Bing Webmaster Tools submission |
What is sitemap.xml?
sitemap.xml is an XML file that lists the URLs on a website a search engine should know about, optionally with the date each one last changed. It is the most common format of the Sitemaps protocol, which Google introduced in 2005 and which Google, Yahoo and Microsoft jointly adopted in 2006 as version 0.9, published at sitemaps.org. The protocol has stayed at 0.9 ever since; it is maintained informally by the search engines rather than by a standards body.
A sitemap does not make pages rank and does not force them into the index. It is a discovery aid: it tells crawlers that a URL exists and, if you are honest about lastmod, when it is worth re-fetching.
How a sitemap works
A minimal sitemap looks like this:
```
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/products/blue-widget</loc>
<lastmod>2026-11-10</lastmod>
</url>
</urlset>
```
The rules that matter in practice:
sitemap.xml.gz), but the 50 MB limit applies to the uncompressed size.<sitemapindex> file lists other sitemaps instead of pages, so one URL can point to hundreds of thousands of pages. Big sites typically split by type or date: sitemap-products-1.xml, sitemap-blog.xml and so on.<loc> must be a full URL on the same host as the sitemap (cross-host sitemaps need verification in Search Console).<priority> and <changefreq>, and Bing has said the same.Sitemap: line in robots.txt, through submission in Google Search Console or Bing Webmaster Tools, or at conventional paths such as /sitemap.xml. Google retired its anonymous sitemap "ping" endpoint in 2023, so submission or the robots.txt line are the routes that remain.The protocol also allows plain-text sitemaps (one URL per line) and RSS or Atom feeds, and Google supports extensions for images, videos and news.
What people use sitemaps for
Site owners use them to get new and changed pages crawled sooner, particularly on large sites, sites with few internal links, or sites that publish often. Search Console's sitemap report is also a quick health check: it shows how many submitted URLs Google has indexed.
People building scrapers and monitors use them the other way around. A sitemap is often the cleanest list of every product, article or listing a site has, published by the site itself. Reading it is cheaper and politer than crawling every link, it avoids pagination entirely, and lastmod lets you fetch only what changed since your last run. For a web crawler, the sitemap is usually the first thing to read after robots.txt.
Common problems
lastmod to today on each build teaches Google to ignore the field for your whole site.noindex pages or non-canonical duplicates. Search Console flags these and they waste crawl attention.&), or the whole file fails to parse.Sitemaps in automation
Sitemaps make a good trigger for change-detection workflows: fetch the sitemap on a schedule, compare it against the last copy, and act on any URL that is new or has a newer lastmod. Because the sitemap is plain XML, this needs only an HTTP request and a parser, not a browser, and it adds little load to the site.
In Autonoly, the built-in "A web page changes" monitoring trigger does a plain HTTP fetch, so it can watch a sitemap URL directly and start a workflow when the file changes. A scheduled workflow can also fetch a sitemap with the HTTP Request step and pass the XML to a later step for parsing. For the pages themselves, especially ones that need JavaScript to render, ask the AI agent to read the sitemap and collect data from the URLs it lists in a real browser.
Why It Matters
A sitemap is the site's own list of the pages it wants found. For SEO it speeds up discovery of new and changed pages; for scraping and monitoring it is often the cleanest, cheapest and most polite way to enumerate every product or article a site has.
How Autonoly Solves It
Autonoly's 'A web page changes' trigger does a plain HTTP fetch, so it can watch a sitemap URL and start a workflow when it changes, and the HTTP Request step can fetch a sitemap on a schedule. The AI agent can read a sitemap and visit the pages it lists in a real browser to collect data.
Learn moreExamples
Splitting a 180,000-product catalog across four sitemap files referenced from one sitemap index
A daily job that fetches a competitor's product sitemap, finds URLs with a newer lastmod than yesterday, and scrapes only those pages
Using Search Console's sitemap report to spot that half the submitted URLs are redirects that should not be listed
Frequently Asked Questions
How many URLs can a sitemap have?
A single sitemap file can list up to 50,000 URLs and be up to 50 MB uncompressed. If you have more, split them into several sitemaps and list those in a sitemap index file, which you then submit or reference from robots.txt.
Does Google use priority and changefreq?
No. Google's documentation says it ignores both. It uses lastmod only when the dates are consistently and verifiably accurate, so update lastmod when the page's main content actually changes, not on every rebuild. Bing has said it ignores priority and changefreq too.
Where should sitemap.xml be located?
Conventionally at the site root, e.g. https://example.com/sitemap.xml, though any location works if you reference it. A sitemap can only list URLs at or below its own path on the same host unless you verify ownership in Search Console. Add a 'Sitemap:' line with the full URL to robots.txt so all crawlers can find it.
Do I need a sitemap?
Small sites with good internal linking often do fine without one. Sitemaps help most on large sites, new sites with few external links, sites that change often, and sites with pages that are hard to reach through navigation. They do not improve rankings directly.
How do I find a website's sitemap?
Open the site's /robots.txt and look for 'Sitemap:' lines. If there are none, try /sitemap.xml and /sitemap_index.xml, which many CMSs use by default. Large sites usually return an index that points to several child sitemaps.
Can I use a sitemap for web scraping?
Yes, and it is often the best starting point. The sitemap lists the URLs the site wants discovered, so you avoid crawling links and paginating through listings, and lastmod lets you fetch only what changed. It may not be complete, and the site's terms and robots.txt still apply.
Further reading
Related terms, automations and guides
Where this concept shows up in practice.
Stop reading about automation.
Start automating.
Describe what you need in plain English. Autonoly's AI agent builds and runs the automation for you — no code required.