Web Crawler

Background. A crawler starts at one page of a website, reads the links on it, follows the ones that stay on the same site, and keeps going until no new page turns up. Here a page is read by calling a function, and site_stub.py holds a small website in memory to try it on.

Implement collect_pages(entry_url, fetch) in site_crawler.py. Return the set of pages found, entry_url included.

• fetch(url) returns the links on the page at url as a list of strings, or raises FetchError (from site_stub.py) when the page cannot be loaded.
• Only pages on the host of entry_url belong to the crawl. A URL's host is the text after http:// or https:// up to the next /, or to the end when there is none, and it must equal the entry's host exactly: blog.atlas.test and atlas.testing are not atlas.test. A page on another host is never fetched.
• Pages link to each other in cycles, and the same link can appear many times. Each page is fetched at most once.
• First cut every URL, entry_url and each link, at its first #, and use only what is left: http://atlas.test/maps#top is the page http://atlas.test/maps. Nothing else about a URL changes: http://atlas.test and http://atlas.test/ are two different pages, and so are http://atlas.test/maps and https://atlas.test/maps, both on the host atlas.test.
• A string that does not start with http:// or https:// is not a link. Skip it; it is not an error.
• When fetch raises FetchError, the page still counts as found, it has no links to follow, and the crawl goes on.
• fetch is called exactly once for every page in the result, and never for anything else.

Example

site = StubSite({
    "http://atlas.test/": ["http://atlas.test/maps", "http://atlas.test/about#team", "http://shop.test/"],
    "http://atlas.test/maps": ["http://atlas.test/", "http://atlas.test/maps/north", "http://atlas.test/maps"],
    "http://atlas.test/about": ["http://atlas.test/maps", "mailto:team@atlas.test"],
    "http://atlas.test/maps/north": ["http://atlas.test/maps#top", "http://atlas.test/"],
    "http://shop.test/": ["http://atlas.test/sale"],
})
collect_pages("http://atlas.test/", site.fetch)
# {"http://atlas.test/", "http://atlas.test/maps", "http://atlas.test/about", "http://atlas.test/maps/north"}

http://shop.test/ is on another host, so it is never fetched, and http://atlas.test/sale, linked only from there, is never found. site.calls lists each of the four pages once.

Notes

• Every URL in the tests is http:// or https://, then a lowercase host (letters, digits, dots, hyphens), then optionally a path starting with /, then optionally # and a fragment. No ports, user names or queries.
• A site has up to about 2000 pages. Following links from the entry without visiting any page twice never passes through more than 400 pages.
• The result may be any collection without repeats; the tests compare it as a set.
• All ten tests in tests/test_part1.py fail on the starter.

Done when Submit goes green.