Design resumable file uploads
Picture
The app creates an authorized upload session. The client sends parts directly to object storage using narrowly scoped access.
Predict
Should a file appear ready merely because an upload session was created?
Walk through the small example
No. Readiness follows verified completion, not intent to upload.
Mental model and runnable Python model
Create a pending metadata record with owner, intended object key, size limits, and expiry. Issue bounded upload capability for that exact scope. The client uploads parts; completion verifies storage state, declared limits, and an appropriate integrity checksum. Do not assume every ETag is a whole-file MD5. Virus/content scanning may keep a file quarantined before ready. Make completion idempotent and reconcile object-store and metadata failures. Lifecycle cleanup must not delete actively referenced objects.
Session API and data
Create an upload session with trusted owner, object key, allowed size/type, expiry, and status. Expose part-signing or provider-supported upload instructions, completion, and cancellation. Keep storage credentials server-side; signed capabilities should be narrow and temporary.
Upload and completion
Send parts directly to object storage. Record/check part identifiers and provider-supported integrity metadata. Completion verifies the assembled object and transitions metadata atomically to verified or quarantined; publish ready only after required scanning succeeds.
Download path
Authorize the current requester against metadata before issuing private access. Use versioned object keys and appropriate CDN behavior. A hash identifies content but does not grant permission, and shared caching must not expose private bytes.
Recovery and cleanup
Retry parts and completion idempotently. Reconcile object-complete/metadata-pending and metadata-created/object-missing cases. Expire abandoned sessions and parts, audit orphan cleanup, and avoid deleting objects still referenced by active records.
What the function promises: UploadSession models numbered byte parts and SHA-256 verification entirely in memory. It does not generate signed URLs, contact R2, or scan files.
from hashlib import sha256
class UploadSession:
def __init__(self, total_parts, expected_hash):
if total_parts < 1:
raise ValueError("Need parts")
self.total_parts, self.expected_hash = total_parts, expected_hash
self.parts, self.result = {}, None
def put_part(self, number, data):
if self.result is not None or not 1 <= number <= self.total_parts:
raise ValueError("Invalid upload state")
data = bytes(data)
if number in self.parts and self.parts[number] != data:
raise ValueError("Conflicting part retry")
self.parts[number] = data
def complete(self):
if self.result is not None:
return self.result
if len(self.parts) != self.total_parts:
raise ValueError("Missing parts")
data = b"".join(self.parts[i] for i in range(1, self.total_parts + 1))
if sha256(data).hexdigest() != self.expected_hash:
raise ValueError("Integrity check failed")
self.result = data
return data
session = UploadSession(2, sha256(b"hello").hexdigest())
session.put_part(2, b"llo")
session.put_part(1, b"he")
session.put_part(1, b"he")
assert session.complete() == b"hello"
assert session.complete() == b"hello"
Work and storage: O(total bytes) completion and memory in this model. Real multipart storage avoids holding the entire upload in application memory.
When to recognize it: Verify all required parts and integrity before marking metadata ready. Uploading bytes and publishing a usable file are different states.
Check before applying it: Upload authorization, byte integrity, malware checks, and metadata readiness are separate checks.
Change one rule
A client claims a smaller size than the bytes actually uploaded. Is trusting its metadata enough?
See what changes
No. Enforce provider-side limits where supported and verify actual stored size and type at completion. Client declarations are untrusted input.
Connect
Separate metadata from large bytes: Object storage holds bytes while a database tracks ownership and readiness.
Commit the change and its event together: A completion event needs durable publication intent alongside metadata changes.