Crawlability check

Robots.txt and Sitemap Validator

Audit robots groups, crawl blocks, declared and default sitemaps, XML namespaces, redirects, URL validity, duplicates, dates, and protocol limits.

SEO Robots + Sitemap Find crawl blocking issues before search engines do.
User-agent: *Provide a default block for general crawlers.
List sitemapAdd your sitemap URL inside robots.txt.
Avoid Disallow: /Use it only when you intentionally want to block all crawling.