Sitemaps generally include the pages you want search engines to crawl and index.
This directive is found at the beginning or end of the robots.txt file and looks like this:
robots.txt sitemap example
That said, you can (and should) submit your denmark number for whatsapp XML sitemap to each search engine using their webmaster tools.
Search engines will eventually crawl your site, but submitting a sitemap speeds up the crawling process.
If you don't want to do this, adding a "Sitemap" directive to your robots.txt file is a quick and easy alternative.
The "crawl-delay" directive specifies a crawl delay in seconds. It is intended to prevent crawlers from overloading the server (i.e. slowing down your website).
However, Google no longer supports this directive.
If you want to set the crawl rate for Googlebot, you will need to do so in Search Console .
Bing and Yandex , on the other hand, support the crawl-delay directive.
Here's how to use it.
If you want a crawler to wait 10 seconds after each crawl action, you can set a 10 second delay like this:
User-agent: *
Crawl-delay: 10
The Noindex Directive
The robots.txt file tells a bot what it can and cannot crawl, but it cannot tell a search engine which URLs not to index and show in search results.
The page will still appear in search results, but the bot won't know what's on it, so your page will look like this:
