cconnected.
Understand the pattern. Find your own way.Find your starting point
Your learning feed/Idea 215 of 218
Ideas this builds on
High-level design215 / 218

Design search and autocomplete

Picture

A primary store keeps documents. An inverted index maps each token to documents containing it, making token lookup efficient.

Predict

Does updating the source document automatically remove its old search terms?

Walk through the small example

No, maintain or rebuild the derived index. The search index is another representation that needs an update protocol.

Mental model and runnable Python model

Define query behavior: prefix suggestions, token matching, language handling, ranking, freshness, and permissions. An inverted index stores postings per term; a document-to-terms view supports updates and deletion. Ranking may combine lexical relevance, recency, popularity, and personalization according to measurable product goals. Index changes through durable events or CDC, use document versions to reject stale updates, and plan rebuild/backfill with a cutover version. Sharding and merging introduce ranking and latency tradeoffs; hot queries can be cached only with correct visibility semantics.

Contract and data

Define token matching, prefix suggestions, language support, ranking, freshness, and visibility. Keep authoritative documents in the source store. The index contains token postings, document versions, ranking fields, and only the metadata needed for retrieval.

Indexing flow

A source commit emits a durable versioned change. Index workers tokenize, remove obsolete postings, add current ones, and record the version. Deletions need tombstones or version rules so delayed events do not resurrect documents.

Query flow

Validate and normalize the query, retrieve candidate postings, combine terms, rank candidates, enforce trusted visibility, and return a bounded page. Prefix suggestions can use a trie or dedicated prefix index; they do not replace full-document retrieval.

Scale and recovery

Partition index data according to query and document distribution, merge ranked shard results, and monitor indexing lag. Rebuild into a new index version, validate it, then switch reads. Cache hot queries only with correct permission and freshness keys.

What the function promises: SearchIndex is a local lowercase-token AND index with replace and delete support. It does not implement production ranking, stemming, distributed indexing, or access control.

import re

class SearchIndex:
    def __init__(self):
        self.postings, self.terms = {}, {}
    def delete(self, doc_id):
        for term in self.terms.pop(doc_id, set()):
            self.postings[term].discard(doc_id)
            if not self.postings[term]:
                del self.postings[term]
    def put(self, doc_id, text):
        self.delete(doc_id)
        terms = set(re.findall(r"\w+", text.casefold()))
        self.terms[doc_id] = terms
        for term in terms:
            self.postings.setdefault(term, set()).add(doc_id)
    def search(self, query):
        terms = set(re.findall(r"\w+", query.casefold()))
        if not terms:
            return []
        groups = [self.postings.get(term, set()) for term in terms]
        return sorted(set.intersection(*groups))

index = SearchIndex()
index.put("a", "tree graph")
index.put("b", "graph search")
assert index.search("graph") == ["a", "b"]
index.put("a", "trie")
assert index.search("tree") == []
assert index.search("graph search") == ["b"]

Work and storage: Updates touch the document’s terms; query cost depends on posting sizes and intersection strategy. This simple version does not optimize term order or compressed postings.

When to recognize it: A trie or prefix index suggests beginnings. Full search also needs tokenization, matching, ranking, and result permissions.

Check before applying it: A search index is a versioned derived view, not an automatically current or authorized copy of the source.

Change one rule

An older update event arrives after a newer one. Should the index blindly apply it?

See what changes

No. Track source versions or another ordering rule per document so stale events cannot resurrect deleted or outdated content.

Connect

tries: A trie shares prefixes; an inverted index maps terms to documents.

autocomplete: Autocomplete adds ranking and bounded suggestions to prefix lookup.