What is robots.txt?

Technically, robots.txt is advisory — not legally binding. However, ignoring it can get your IP blocked, trigger legal action (some courts consider robots.txt in scraping lawsuits), and harm your reputation. Ethical scrapers respect robots.txt unless they have explicit permission from the site owner.

No. robots.txt only works for bots that choose to obey it. Malicious bots and most web scrapers can still access disallowed paths. It is a polite request, not a security measure. For actual access control, use authentication, firewalls, or rate limiting.

robots.txt can prevent Googlebot from crawling a page, but if other sites link to that page, Google may still index the URL (showing it in results without a snippet). To fully remove a page from search results, use a "noindex" meta tag or X-Robots-Tag HTTP header instead.

robots.txt must be at the root of your domain: https://example.com/robots.txt. It only applies to that specific domain and protocol. A robots.txt on example.com does not affect subdomain.example.com — each needs its own file.

robots.txt tells scrapers which paths the site owner prefers not to be crawled. Ethical scrapers check robots.txt before requesting pages. SnapRender respects robots.txt by default but lets you override this when you have permission to scrape restricted paths.