Crawling is directly related to indexing. To optimize indexing, you can guide Google on how to best crawl your site.


xml format sitemap guides Google on how to crawl your site. Although Google says there is no guarantee the Google bot will follow your instructions, it is still highly recommended and in most cases, sitemaps are beneficial. It contains useful information about each page on your site, including when a page was last modified; what priority it has on your site; how frequently it is updated.

Search Console Sitemap


Robots set instructions depending on the user agent which parts of the site can be accessed. Making sure you are not excluding the relevant search engine bots is therefore of course paramount.


You can also guide the Google bot in the code with tags for each page the most common ones being: follow/ nofollow and index/ noindex

Search Console

You can directly submit urls to the Google index in the Google Search Console. This is particularly useful if you have crawling issues and there are some pages you want to have crawl and indexed in priority.

Search Console Crawling
Search Console Crawlability

Indexation Google Crawl Credit

An important factor which links crawling and indexation is that for sites with many pages (indexed or not), there is a limit to how much your site will be crawled each time the Google bot visits your site. It is therefore important to keep an eye and understand which pages are indexed and why they need to be indexed (not all pages need to be indexed in particular in case of duplicate content)

Alex Bieth, Owner @ SEO Berlino and SEO Consultant