cconnected.
Understand the pattern. Find your own way.Find your starting point
Your learning feed/Idea 213 of 218
Ideas this builds on
High-level design213 / 218

Design resumable file uploads

Picture

The app creates an authorized upload session. The client sends parts directly to object storage using narrowly scoped access.

Predict

Should a file appear ready merely because an upload session was created?

Walk through the small example

No. Readiness follows verified completion, not intent to upload.

Mental model and runnable Python model

Create a pending metadata record with owner, intended object key, size limits, and expiry. Issue bounded upload capability for that exact scope. The client uploads parts; completion verifies storage state, declared limits, and an appropriate integrity checksum. Do not assume every ETag is a whole-file MD5. Virus/content scanning may keep a file quarantined before ready. Make completion idempotent and reconcile object-store and metadata failures. Lifecycle cleanup must not delete actively referenced objects.

Session API and data

Create an upload session with trusted owner, object key, allowed size/type, expiry, and status. Expose part-signing or provider-supported upload instructions, completion, and cancellation. Keep storage credentials server-side; signed capabilities should be narrow and temporary.

Upload and completion

Send parts directly to object storage. Record/check part identifiers and provider-supported integrity metadata. Completion verifies the assembled object and transitions metadata atomically to verified or quarantined; publish ready only after required scanning succeeds.

Download path

Authorize the current requester against metadata before issuing private access. Use versioned object keys and appropriate CDN behavior. A hash identifies content but does not grant permission, and shared caching must not expose private bytes.

Recovery and cleanup

Retry parts and completion idempotently. Reconcile object-complete/metadata-pending and metadata-created/object-missing cases. Expire abandoned sessions and parts, audit orphan cleanup, and avoid deleting objects still referenced by active records.

What the function promises: UploadSession models numbered byte parts and SHA-256 verification entirely in memory. It does not generate signed URLs, contact R2, or scan files.

from hashlib import sha256

class UploadSession:
    def __init__(self, total_parts, expected_hash):
        if total_parts < 1:
            raise ValueError("Need parts")
        self.total_parts, self.expected_hash = total_parts, expected_hash
        self.parts, self.result = {}, None
    def put_part(self, number, data):
        if self.result is not None or not 1 <= number <= self.total_parts:
            raise ValueError("Invalid upload state")
        data = bytes(data)
        if number in self.parts and self.parts[number] != data:
            raise ValueError("Conflicting part retry")
        self.parts[number] = data
    def complete(self):
        if self.result is not None:
            return self.result
        if len(self.parts) != self.total_parts:
            raise ValueError("Missing parts")
        data = b"".join(self.parts[i] for i in range(1, self.total_parts + 1))
        if sha256(data).hexdigest() != self.expected_hash:
            raise ValueError("Integrity check failed")
        self.result = data
        return data

session = UploadSession(2, sha256(b"hello").hexdigest())
session.put_part(2, b"llo")
session.put_part(1, b"he")
session.put_part(1, b"he")
assert session.complete() == b"hello"
assert session.complete() == b"hello"

Work and storage: O(total bytes) completion and memory in this model. Real multipart storage avoids holding the entire upload in application memory.

When to recognize it: Verify all required parts and integrity before marking metadata ready. Uploading bytes and publishing a usable file are different states.

Check before applying it: Upload authorization, byte integrity, malware checks, and metadata readiness are separate checks.

Change one rule

A client claims a smaller size than the bytes actually uploaded. Is trusting its metadata enough?

See what changes

No. Enforce provider-side limits where supported and verify actual stored size and type at completion. Client declarations are untrusted input.

Connect

Separate metadata from large bytes: Object storage holds bytes while a database tracks ownership and readiness.

Commit the change and its event together: A completion event needs durable publication intent alongside metadata changes.