Site Validator (robots, sitemap, SSL, headers)

Comprehensive technical check: robots.txt parsing, sitemap discovery and URL count, SSL certificate issuer and expiry, security headers (HSTS, CSP, X-Frame-Options, Referrer-Policy), and server info — all in one click. Hosting with us? Run it before you launch. Hosting elsewhere? Run it to see what your host isn't configuring.

5 / 5 free runs left today · Sign up free for 100/day

About the Site Validator (robots, sitemap, SSL, headers)

This is your one-click technical SEO health check. It validates four things at once: robots.txt (parsing directives, extracting sitemaps), sitemap.xml (discovery and URL count), SSL certificate (issuer, validity, days until expiry), and security HTTP headers (HSTS, CSP, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy). Get everything green here and both Google crawlers and security scanners will be happy with your site.

How to read your results

Robots.txt

Your robots.txt file lives at yoursite.com/robots.txt and tells crawlers what they can and cannot access. The tool confirms it exists, counts how many sitemap references it contains, and lists the first 50 Disallow rules. If robots.txt is missing, Googlebot assumes everything is crawlable — which is fine but means you have no control over crawl budget.

Sitemap.xml

Your sitemap tells Google every URL you want indexed. The tool tries the URL in robots.txt first, then /sitemap.xml, then /sitemap_index.xml. A healthy small-business site has 20–200 URLs in its sitemap; a blog should have every post. An empty or missing sitemap is the #1 reason new pages take weeks to appear in Google.

SSL Certificate

The tool connects to port 443, reads the certificate, and reports the issuer, validity period, and days remaining. Certs expiring in under 30 days show a warning. The most common failure is an expired Let's Encrypt cert because auto-renewal broke silently — this tool will catch it before your traffic does.

Security Headers

Modern browsers reward sites with proper security headers, and many security scanners penalize sites without them. The six headers we check are the core set from OWASP. You do not need every single one, but HSTS and X-Content-Type-Options are non-negotiable. Content-Security-Policy is the hardest to set up and the most impactful.

Frequently asked questions

My robots.txt is missing — is that a problem?

Not urgently, but you should add one. Without robots.txt, Googlebot crawls everything it can find, which wastes crawl budget on admin pages, search result pages, and duplicate URLs. A good baseline robots.txt: Allow: /, Disallow: /admin/, Disallow: /search?, and a Sitemap: line pointing to your sitemap.xml.

My sitemap has 50 URLs but my site has 500. Why?

Your CMS is probably only adding published posts or top-level pages to the sitemap. Check your SEO plugin settings — Yoast, Rank Math, and similar plugins all have toggles for which content types appear in the sitemap. Also verify category/tag/archive pages are included if you want them indexed.

My SSL cert has 14 days left — what do I do?

If you use Let's Encrypt (most hosts do), auto-renewal has broken. SSH into your server and manually trigger renewal with "certbot renew" or your host's equivalent command. Then fix the cron job that was supposed to run it. If you are on a managed host (Cloudflare, Netlify, Vercel), this is almost certainly a config error on their side — open a support ticket.

Do I really need all 6 security headers?

The four essential ones: HSTS, X-Content-Type-Options, X-Frame-Options, and Referrer-Policy. CSP is extremely powerful but takes real effort to deploy without breaking third-party scripts. Permissions-Policy is new and mostly affects browser feature APIs most sites do not use. Start with the four essentials.

My HSTS header has a short max-age — should I increase it?

Yes. The recommended HSTS max-age is 31536000 (1 year) once you are confident every subresource on your site is reachable over HTTPS. Start with a short max-age during rollout, then bump to 1 year. Add "includeSubDomains" and "preload" only when you are fully committed — unsetting preload is painful.

The tool says my site has CSP "present" but my developer says we do not use it. Why?

Your hosting provider, CDN, or a reverse proxy is injecting one. Cloudflare does this for some plans. Check your origin server's raw response with curl -I to see whether the header originates from your app or from a layer above. This is fine — just be aware of it if you ever need to tighten the policy.

My server info shows "Apache" — should I hide it?

Security-through-obscurity is not real security, but yes, hiding server banners is a cheap win. It takes attackers one extra step to find the exploit matching your specific server version. In Apache, set "ServerTokens Prod" and "ServerSignature Off". In Nginx, set "server_tokens off". Neither changes your security posture but both reduce noise in automated scan reports.

Can I save or share these results?

Not yet as a shareable link, but you can screenshot or print the page directly. We are adding permalink support to tool results so you can share a findings URL with your dev team or client.

Related tools, articles & authoritative sources

Hand-picked internal pages and external references from sources Google itself considers authoritative on this topic.