# EternitySong — robots.txt
#
# EVERY group must repeat the /admin/ disallow. A crawler obeys only the single
# most specific User-agent group that matches it, so naming a crawler makes it
# ignore the wildcard group ENTIRELY — including that group's Disallow.
#
# The previous version listed Googlebot, Bingbot, Twitterbot and
# facebookexternalhit with a bare "Allow: /", and put "Disallow: /admin/" only
# under "User-agent: *". Net effect: all four of those crawlers were explicitly
# invited into /admin/. (It also declared "User-agent: *" twice, which is
# malformed — behaviour on duplicate groups varies by crawler.)
#
# NOTE ON THE SONG PAGES — /song/, /q/ and /v/ are deliberately NOT disallowed
# here, and that is not an oversight. Those pages already send
# <meta name="robots" content="noindex,nofollow">. Disallowing them in robots
# would be counter-productive: a disallowed URL can still be indexed URL-only if
# someone links to it publicly, and because the crawler is forbidden from
# fetching the page it can never SEE the noindex telling it not to. Letting the
# crawl through is what actually keeps a shared song out of the index.

User-agent: Googlebot
Allow: /
Disallow: /admin/

User-agent: Bingbot
Allow: /
Disallow: /admin/

User-agent: Twitterbot
Allow: /
Disallow: /admin/

User-agent: facebookexternalhit
Allow: /
Disallow: /admin/

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://eternitysong.com/sitemap.xml