What is the algorithm for deciding when to not crawl a page anymore?

How does the crawler decide that a page, or a whole website, no longer exists and should be dropped from the list of things to be crawled? For instance, if the DNS for a server hostname stops resolvi...

Posted by: zwol