In order to avoid unwanted content from the look for indexes, site owners can instruct spiders to not crawl certain data files or directories with the normal robots.txt file in the foundation Listing from the domain. Moreover, a page can be explicitly excluded from a online search engine's databases by making use of a meta tag precise to robots (or