Iterator: reveal items one at a time
Picture
A page has two notes. Ask for the next page only when those notes have been consumed.
Predict
Does returning an iterator automatically make a remote dataset a consistent snapshot?
Walk through the small example
No. Snapshot semantics need support from the data source and query contract.
Mental model and complete Python solution
Iterator separates traversal from representation. page_items requests a page, yields its items, and follows the returned cursor. The generator stores only the current page and cursor rather than collecting all data. Its fetch collaborator can be replaced in tests. The contract requires cursors to advance and eventually terminate; otherwise malformed pagination could loop forever.
What the function promises: fetch(cursor) returns (items, next_cursor), with None marking the end. The sample uses a fixed in-memory dataset and never performs network calls.
def page_items(fetch):
cursor = None
while True:
items, following = fetch(cursor)
yield from items
if following is None:
return
if following == cursor:
raise ValueError("Pagination cursor did not advance")
cursor = following
pages = {None: (["a", "b"], "second"),
"second": (["c"], None)}
calls = []
def fetch(cursor):
calls.append(cursor)
return pages[cursor]
items = page_items(fetch)
assert next(items) == "a" and calls == [None]
assert list(items) == ["b", "c"]
assert calls == [None, "second"]
Work and storage: O(total items + page-fetch work) time and O(page size) local storage, excluding the data source.
When to recognize it: A cursor can move through a stable order, but pagination alone does not promise a snapshot while concurrent writes occur.
Check before applying it: Lazy iteration does not define ordering, snapshot consistency, or resource cleanup by itself.
Change one rule
A client stops iterating after the first result. Must every remaining page be fetched?
See what changes
No. Lazy iteration fetches only as needed. If a real fetch holds connections or streams open, also provide explicit cleanup or a context manager.
Connect
loops: A loop can consume one item at a time without knowing storage.
Make an API safe to call again: Remote cursors add ordering, failure, and consistency requirements.