Question about visibility
N6ghost
n6ghost at gmail.com
Sun Oct 21 19:32:06 UTC 2018
On Thu, 11 Oct 2018 15:39:55 -0400
Barry Margolin <barmar at alum.mit.edu> wrote:
> In article <mailman.671.1539286015.803.bind-users at lists.isc.org>,
> Dennis Clarke <dclarke at blastwave.org> wrote:
>
> > On 10/11/2018 03:21 PM, Leonardo Rodrigues wrote:
> > > Em 11/10/18 16:13, Barry Margolin escreveu:
> > >>
> > >> If you accidentally, or someone else intentionally, create a
> > >> link to the site that uses the IP and put it on a web page that
> > >> Google can get to, it will probably find the page.
> > >>
> > >>
> > >
> > > robots.txt, on your website root, is your friend. Simply
> > > deny web crawling on it, and you're (probably) done.
> > >
> >
> > If you believe robots.txt means anything at all.
>
> Google is known to obey it, and the question was about avoiding
> getting your site indexed by Google.
>
> Of course, that doesn't mean someone won't find the site on their
> own. If the link to it is on some other page that isn't blocked by
> robots.txt, someone might stuble across that page and then click on
> the link.
>
> But if you're mainly worried about someone googling the words that
> are on your website and Google sending them to the development
> version instead of the production version, you're pretty safe.
>
> Actually, DNS has very little impact on this at all. AFAIK, Google
> doesn't crawl DNS, it just crawls web pages and follows links. My
> company's development server is in DNS, and it's not firewalled (we
> all work from our homes, there's no company network to restrict
> access with), but I've never heard of anyone accidentally being
> directed there by Google, because we don't publish links to this
> server.
>
robot.txt is suppose to govern whats indexed... not sure how well its
followed nowadays but thats the process for it.
More information about the bind-users
mailing list