[lib-storage] Upload.done() never resolves after large streaming upload (unknown size)

[fooman5000]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

When using Upload from @aws-sdk/lib-storage with a streaming body of unknown size (no ContentLength), after all data has been written and the stream is ended, upload.done() can never resolve. The multipart upload completes successfully on S3 (all parts uploaded, object is usable), but the promise from done() never settles, so the application hangs until manually stopped.
This is different from #5652 (Upload stuck with empty file readable stream). Our case is large streaming payloads (e.g. 35–45 GB) from a PassThrough (or similar) stream where total size is unknown up front.

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/client-s3@3.621.0, @aws-sdk/lib-storage@3.490.0 (also observed in same scenario with other 3.x versions).

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

Node.js 20.x (LTS)

Reproduction Steps

Create a PassThrough (or other readable) stream as the upload body.
Create Upload with { client, params: { Bucket, Key, Body: stream } } — do not set ContentLength (streaming/unknown size).
Start the upload with upload.done() and write a large amount of data to the stream (e.g. tens of GB) over time.
When the export is finished, call stream.end() so the stream closes.
Await upload.done() (e.g. in a “close” or “shutdown” path).

Observed Behavior

The process hangs at the await upload.done() step. No resolve or reject. The object in S3 is complete and correct; only the promise never settles.

Expected Behavior

done() should resolve with CompleteMultipartUploadCommandOutput (or reject with an error) once the stream has ended and the final CompleteMultipartUpload has been sent and responded to.

Possible Solution

Wrapping upload.done() in a timeout (e.g. Promise.race with a timer) so the application does not block indefinitely. This avoids the hang but does not fix the underlying behavior.

Additional Information/Context

We use the recommended pattern: Upload from lib-storage with a stream body for large/streaming uploads.
We do not pass checksums (e.g. ChecksumSHA256).
Occurrence is intermittent; not every large streaming run hangs, but when it does, it is consistently at the same point (after stream end, waiting on done()).

[fooman5000]

Similar behavior (multipart upload hanging / no error) is reported in AWS re:Post: AWS S3 multipart upload hangs up, and in Why do S3 CLI uploads sometimes hang mid-transfer with no error?. This issue is the SDK/Node.js case: upload.done() never resolves after stream ends.

[fooman5000]

Side note: this conversation documents that the client can be left waiting when the response to CompleteMultipartUpload never arrives, and that S3 may already have finished the merge.

[aBurmeseDev]

Can you share your code snippet? I'm unable to reproduce this with code below.

Also are you seeing with both JS SDK and AWS CLI since your comments mentioned links to issues with CLI related?

const { S3Client } = require("@aws-sdk/client-s3");
const { Upload } = require("@aws-sdk/lib-storage");
const { PassThrough } = require("stream");

const BUCKET = "your-test-bucket";
const KEY = `test-upload-${Date.now()}.bin`;
const CHUNK_SIZE = 10 * 1024 * 1024; 
const TOTAL_SIZE = 100 * 1024 * 1024; 

async function upload() {
  const client = new S3Client({ region: "us-east-1" });
  const stream = new PassThrough();

  const upload = new Upload({
    client,
    params: {
      Bucket: BUCKET,
      Key: KEY,
      Body: stream,
    },
  });

  const uploadPromise = upload.done();

  let bytesWritten = 0;
  const writeInterval = setInterval(() => {
    if (bytesWritten >= TOTAL_SIZE) {
      clearInterval(writeInterval);
      stream.end();
      return;
    }

    const chunk = Buffer.alloc(CHUNK_SIZE, "x");
    stream.write(chunk);
    bytesWritten += CHUNK_SIZE;
  }, 100);

  // Add timeout to detect hang
  const timeout = setTimeout(() => {
    process.exit(1);
  }, 120000); // 2 minutes

  try {
    console.log("Awaiting upload.done()...");
    const result = await uploadPromise;
    clearTimeout(timeout);
    console.log("SUCCESS: upload.done() resolved");
    console.log("Result:", {
      Bucket: result.Bucket,
      Key: result.Key,
      ETag: result.ETag,
      Location: result.Location,
    });
  } catch (error) {
    clearTimeout(timeout);
    console.error("ERROR: upload.done() rejected with error:", error);
    process.exit(1);
  }
}

upload().catch((error) => {
  console.error("Unhandled error:", error);
  process.exit(1);
});

[paffoobar]

Running further tests, this issue is only happening when sending 40GB+ files. We noticed that the hang does not occur if the CompleteMultipartUpload takes less than 15 minutes. There is no apparent proxy in between us and AWS, but it "seems" like a transparent proxy is closing the connection. The solution seems to be to add keep alives in our client http agent. It seems the S3 client may not send these while the server side is stitching together a large file. If this is the case, an enhancement to the client might be to automatically send keep alives to avoid middleware (DLP or transparent proxy) from resetting the connection.

keepAlive: true
keepAliveMsecs: 60000