Skip to content
Browse documentation
On this page

Add your site

What happens when you add a site, which pages get indexed and how to read the indexing status.

TalkToMySites reads your site through its sitemap. Every page listed in the sitemap is fetched, cleaned up and indexed so the assistant can answer from it.

Add a site

Open Sites and fill in the Add a site card:

  • Sitemap URL (required). For example https://example.com/sitemap.xml.
  • Name (optional). Shown in your dashboard. If you leave it blank we use the domain.
  • Widget domain (optional). The domain where the widget is allowed to run. If you leave it blank we use the sitemap's domain. See Install the widget.

Click Add site. Reading a large sitemap can take a few seconds.

Only team owners and admins can add sites.

Sitemap rules

  • The sitemap must be on the site's domain. A sitemap on another domain is refused.
  • Sitemap index files are supported, up to 25 child sitemaps. Gzipped sitemaps (.xml.gz) work too.
  • Only pages on the same domain or its subdomains are kept.
  • The domain must be your own. Shared hosting suffixes such as github.io or vercel.app on their own are refused; use your full subdomain instead, like yourname.github.io.

If you don't know your sitemap URL, try /sitemap.xml or /sitemap-index.xml on your domain. Most site builders and CMSs make one for you.

What gets indexed

Before indexing, we remove navigation, the page header, footers, sidebars, forms, scripts and elements with the hidden attribute. The rest of the page text is kept, so the assistant answers from your content and not your menus.

These are not indexed:

  • Files that aren't web pages, such as PDFs, images, CSV, ZIP and Office documents.
  • Pages with almost no text (under 40 characters after cleanup). They show as Skipped.

Each page is read up to 200,000 characters.

How many pages

Your plan sets the number of pages per site: 50 on Free, 500 on Starter and 2,000 on Pro. If your sitemap lists more, we index up to the limit and leave out the rest. If the Total on the Indexing card equals your plan's limit, some pages were probably left out. See Plans and limits.

Indexing status

The Indexing card on your site page counts pages as Total, Indexed, Pending, Failed and Skipped. It refreshes every few seconds while pages are pending.

Pages are crawled in batches of 10, once a minute, taking turns with other sites that are waiting. Click Process next batch now if you want the next batch straight away.

Open Failed pages or Skipped pages to see the first 20 pages in each list and the reason. A page usually fails because it timed out (we wait 15 seconds) or returned an error.