No. One host can still receive a concentrated burst.

The frontier owns canonical URL identity, priority, retries, and per-host pacing. Discovery parses permitted links from fetched content and enqueues only in-scope candidates. Robots rules are a standard policy input, not authorization to bypass authentication or ignore terms. A real fetcher needs DNS and redirect revalidation, private-address restrictions, content/size limits, timeouts, and abuse protections. Canonicalization should not erase query distinctions blindly. Distributed workers need shared host ownership or coordination so local pacing does not multiply across machines.

Requirements and frontier

Define permitted domains, crawl purpose, freshness, depth, byte budget, and retention. The frontier stores canonical URL, host, priority, attempt count, next eligible time, and status. Use stable deduplication with an explicit revisit policy.

Fetch and discovery

Before fetching, apply policy and per-host scheduling. The fetcher enforces timeouts, redirect limits, public-network restrictions, response-size limits, and supported content types. Parse bounded content and enqueue only permitted in-scope links.

Distributed ownership

Assign hosts to schedulers or coordinate host budgets so many workers do not independently burst against the same site. Keep frontier state durable enough to recover, and separate transient fetch failures from permanent policy rejection.

Limits and safety

Honor robots rules and site access policies; do not bypass authentication. URL normalization, DNS rebinding, redirects, compression bombs, and infinite query spaces require explicit safeguards. This course model schedules URLs but never fetches them.

What the function promises: Frontier schedules only explicitly allowlisted HTTPS hosts and a supplied robots-policy result. It performs no network fetch. It is not a complete SSRF filter or robots.txt parser.

from urllib.parse import urlsplit, urlunsplit

class Frontier:
    def __init__(self, hosts, spacing):
        if spacing < 0:
            raise ValueError("Invalid spacing")
        self.hosts, self.spacing = set(hosts), spacing
        self.seen, self.next_time = set(), {}
    def admit(self, url, now, robots_allows):
        parsed = urlsplit(url)
        host = parsed.hostname
        if (parsed.scheme != "https" or host not in self.hosts
                or parsed.username or parsed.password
                or parsed.port not in (None, 443) or not robots_allows(url)):
            return None
        canonical = urlunsplit(("https", host, parsed.path or "/", parsed.query, ""))
        if canonical in self.seen:
            return None
        at = max(now, self.next_time.get(host, now))
        self.next_time[host] = at + self.spacing
        self.seen.add(canonical)
        return at, canonical

frontier = Frontier({"example.com"}, 2)
assert frontier.admit("https://example.com/a", 0, lambda u: True)[0] == 0
assert frontier.admit("https://example.com/b", 0, lambda u: True)[0] == 2
assert frontier.admit("https://example.com/a#part", 0, lambda u: True) is None
assert frontier.admit("https://localhost/", 0, lambda u: True) is None

Work and storage: Expected O(1) local dedup and host scheduling, excluding URL parsing and real fetch work. Storage grows with discovered URLs and hosts.

When to recognize it: Honor applicable robots rules and access policies. Also prevent fetches into private networks, unsafe redirects, or unexpectedly large responses.

Check before applying it: Robots policy, host pacing, scope, and network-fetch safety are separate responsibilities.

No. The fetch layer must validate resolved destinations and every redirect against network policy, with egress controls and bounded resolution behavior. No fetching occurs in this model.