Chunking Strategy

When uploading massive files (like a 50GB video), we cannot load them entirely into memory. Doing so will immediately invoke the OOM killer, regardless of whether you are running on an iPhone, a Linux desktop, or a Chrome browser tab.

To safely and rapidly process these massive files, Platrium's SDK relies on a parallelized Chunking Engine with built-in concurrency controls and backpressure.

The 4MB Rule

We slice all files into strict 4MB chunks.

pub const CHUNK_SIZE_BYTES: u64 = 4 * 1024 * 1024; // 4MB

Why 4MB? It is the perfect Goldilocks zone for network transfers:

• Small enough to keep our RAM footprint extremely low during IO operations.
• Large enough that we don't spam our backend or object storage with millions of tiny chunk records and API requests.

The ChunkProcessor

The core of our strategy is the ChunkProcessor. It calculates byte offsets on the fly without holding the actual file contents in memory.

Zero-Copy References

The processor takes an immutable reference to our XPlatFile abstraction. By using a lifetime reference &'a XPlatFile, we ensure that the chunking engine never takes ownership of the file handle or wastes CPU cycles trying to clone it.

Parallel Hashing with Concurrency Control

Before uploading chunks, we generate their SHA-256 hashes to ask the server if they already exist. This process is heavily parallelized for performance but tightly throttled to prevent memory spikes.

128-Chunk Batches

Instead of hashing the entire file upfront (which delays the upload from starting) or doing it strictly sequentially, we process the file in batches of 128 chunks (approx. 512MB windows). This drastically reduces the time-to-first-chunk upload. Network requests for the first batch begin immediately while the next batch is being hashed in the background.

Tokio Offloading & CPU-bound Blocking

Hashing is a CPU-bound operation. To avoid starving the async network runtime, the SDK spawns lightweight tokio tasks to orchestrate the I/O, which internally offload the actual SHA-256 math to a dedicated OS thread via tokio::task::spawn_blocking:

handles.push(tokio::spawn(async move {
    // 1. Acquire permit to throttle memory usage
    let _permit = CHUNK_SEMAPHORE.acquire().await?;
    
    // 2. Read exactly 4MB into memory
    let buffer = xplat_file.read_exact_at(offset, size).await?;

    // 3. Offload CPU-heavy SHA-256 hashing to a blocking thread
    let hash = tokio::task::spawn_blocking(move || {
        let mut hasher = Sha256::new();
        hasher.update(&buffer);
        format!("{:x}", hasher.finalize())
    }).await?;
    
    // ...
}));

Chunking Backpressure

Spawning 128 parallel 4MB reads would instantly load 512MB into memory, crashing lower-end mobile devices. To prevent this, we introduce a strictly enforced bottleneck in fs/chunks.rs:

// Allow at most 5 concurrent chunking tasks to limit memory footprint (5 * 4MB = 20MB max overhead)
static CHUNK_SEMAPHORE: Semaphore = Semaphore::const_new(5);

By requiring tasks to acquire a permit from a global tokio::sync::Semaphore before performing the disk read, we mathematically guarantee that our peak memory footprint for chunk reading will never exceed ~20MB.

Global Backpressure

The SDK guarantees stable resource usage across both disk and network:

• Disk/CPU Backpressure: Handled by the CHUNK_SEMAPHORE in the Chunking Engine (limits I/O and memory overhead to 5 concurrent chunks).
• Network Backpressure: Handled by the NetworkTransferManager (limits concurrent active HTTP connections and chunk uploads).

This unified strategy allows the SDK to maximize I/O and Network parallelism, processing gigabytes of data on devices without breaking a sweat.

Performance Benchmarks