# See https://www.robotstxt.org/robotstxt.html for how this file is read.
#
# Everything a reader can open is open to a crawler. What is disallowed here
# is an address with a query string: the ranking filtered, sorted, paged or
# compared, which is thousands of combinations of the same rows, each of which
# a crawler would fetch and none of which is a page of its own. Every chain,
# city and product has an address without one.
#
# The pages that are not a reader's do not appear here. A disallow line is a
# published list of where to look, read by anybody scanning for a way in, and
# it is the wrong tool for the job: those pages are linked from nowhere public
# and every one of them answers a crawler with X-Robots-Tag: noindex, which
# keeps them out of an index without naming them. See ApplicationController.
User-agent: *
Disallow: /compare
Disallow: /*?