Analysis Results
✓ No critical issues
Directives
| # |
Directive |
Value |
User-Agent |
Best Practices
- Use User-agent: * Always provide a default block for all crawlers.
- List sitemaps: Add "Sitemap: https://example.com/sitemap.xml" in robots.txt.
- Keep sitemaps fresh: Update lastmod dates and ensure URLs return HTTP 200.
- Avoid disallowing everything: Don't use "Disallow: /" unless intentional.