Handling 10GB File Uploads in Node.js (Without Multer)

Handling file uploads in Node.js has historically been handled by multer. While multer is great, it connects to Express's middleware chain which can add overhead when dealing with purely streaming data.

When a user uploads a 10GB video file, you want that data to flow from the Request socket directly to the Disk (or S3) socket, touching your RAM as little as possible.

The Streaming Problem

By default, many frameworks try to buffer the "field" parts of a multipart request. If you have a form with metadata AND a file, the order matters.

In Newgate, we solved this with an async iterator for multipart processing.

The Code

Here is how you handle a massive upload in Newgate:

import { pipeline } from 'stream/promises';
import { createWriteStream } from 'fs';

app.post('/upload', async (req, res) => {
    // Newgate detects 'multipart/form-data'
    // and provides an async iterator for parts
    for await (const part of req.body) {
        
        if (part.file) {
            // It's a file stream!
            // Pipe directly to disk
            await pipeline(
                part.value, 
                createWriteStream(`./uploads/${part.filename}`)
            );
        } else {
            // It's a field (e.g., 'uploade_by')
            console.log(`Field ${part.name}: ${part.value}`);
        }
    }

    res.json({ success: true });
});

Why this matters

1. Low Memory Footprint: We never hold the full file in memory.
2. Backpressure: Standard Node.js streams handle backpressure automatically. If the disk is slow, the network socket pauses.
3. Simplicity: No complex storage engine configuration. Just standard Streams.

Comparison with Multer

Start building high-performance storage services today.

Learn more about Binary Parsing