Create robots.txt files with multiple user-agent rules, allow and disallow directives, and sitemap URLs. Includes presets for common crawlers.
Loading tool…