Crawler-correct website template
A reference site implementing Google's web-crawling protocols end to end:
a REP-correct robots.txt, a
sitemap.xml with accurate
<lastmod>, conditional-request HTTP caching
(ETag/Last-Modified), correct status-code handling,
and Googlebot verification tooling.
How to use this template
Replace
example.com throughout, point the server at your content,
and keep the protocol files (robots.txt, sitemap.xml,
error pages, caching headers) intact.