coderain blog

AWS S3 Access Denied Only Sometimes: Troubleshooting Intermittent File Uploads with JavaScript SDK (Multiple Successive Files)

Imagine this: You’ve built a JavaScript application that uploads files to Amazon S3. Most of the time, it works flawlessly—files sail into your bucket without a hitch. But occasionally, especially when uploading multiple files in quick succession, you hit a frustrating AccessDenied error. The inconsistency is maddening: why does it work sometimes but not others?

Intermittent S3 Access Denied errors during multi-file uploads are a common pain point with the AWS SDK for JavaScript, often stemming from the dynamic interplay of credentials, concurrency, CORS, and S3’s strict security model. Unlike consistent errors (e.g., a misconfigured IAM policy), intermittent failures are trickier to diagnose because they depend on timing, network conditions, or state changes (e.g., expiring credentials).

In this blog, we’ll demystify these intermittent errors. We’ll break down the root causes, walk through systematic troubleshooting steps, and share solutions to ensure reliable multi-file uploads with the AWS SDK for JavaScript.

2026-01

Table of Contents#

  1. Understanding Intermittent Access Denied Errors
  2. Common Causes of Intermittent Failures
  3. Troubleshooting Steps: A Systematic Approach
  4. Solutions and Best Practices
  5. Conclusion
  6. References

1. Understanding Intermittent Access Denied Errors#

Intermittent AccessDenied errors occur when S3 rejects a request only under specific conditions, rather than consistently. Unlike static issues (e.g., a missing IAM permission), these failures depend on dynamic factors like:

  • Credential freshness (e.g., temporary tokens expiring mid-upload).
  • Network timing (e.g., delayed CORS preflight responses).
  • Concurrency (e.g., 10 simultaneous uploads overwhelming rate limits).
  • Conditional logic (e.g., IAM policies allowing access only during business hours).

In JavaScript applications, the asynchronous nature of file uploads amplifies these issues. For example, if your code uses Promise.all to upload 5 files at once, race conditions (e.g., credentials expiring before the 3rd upload) can cause sporadic failures.

2. Common Causes of Intermittent Failures#

Let’s dive into the most likely culprits behind intermittent S3 upload errors.

2.1 Temporary Credential Expiry#

If your application uses temporary credentials (e.g., from Amazon Cognito, AWS STS, or IAM Roles Anywhere), these credentials have a finite lifespan (typically 15 minutes to 12 hours). When uploading multiple files, the first few may succeed, but later uploads may fail if the token expires before the SDK refreshes it.

Example Scenario: You fetch a temporary token with a 5-minute expiry, then start uploading 10 large files. The first 7 finish within 5 minutes, but the 8th starts after the token expires—triggering AccessDenied.

2.2 Rate Limiting and Throttling#

S3 enforces request rate limits (e.g., 3,500 PUT requests per second per prefix for standard buckets). If your JavaScript code uploads files too quickly (e.g., 100 concurrent uploads), S3 may throttle some requests. While throttling typically returns 503 Slow Down, misconfigured clients may misinterpret this as AccessDenied.

Example: Using Promise.all to upload 50 files at once exceeds S3’s rate limit. Some requests are throttled, and your SDK (or browser) misclassifies the error as access denial.

2.3 CORS Misconfigurations#

For browser-based JavaScript apps, S3 requires CORS (Cross-Origin Resource Sharing) configuration to allow uploads from your domain. Intermittent failures can occur if:

  • Preflight OPTIONS requests (required for cross-origin POSTs) fail due to network delays or misconfigured headers.
  • The CORS policy allows only specific headers (e.g., Content-Type) but your app occasionally sends additional headers (e.g., x-amz-meta-custom), causing preflight rejection.

2.4 SDK Configuration Issues#

Outdated AWS SDK versions, incorrect regions, or misconfigured endpoints can lead to intermittent failures. For example:

  • Using an older SDK version with a bug in credential refresh logic.
  • Configuring the SDK for us-east-1 but uploading to a bucket in eu-west-1 (S3 returns AccessDenied instead of a region mismatch error in some cases).

2.5 Concurrency and Asynchronous Race Conditions#

JavaScript’s asynchronous model can create race conditions. For example:

  • Your code initiates a credential refresh but starts an upload before the refresh completes, using stale credentials.
  • You reuse an S3 client instance across multiple uploads, but the client’s internal state (e.g., cached credentials) becomes inconsistent.

2.6 Conditional Bucket Policies or IAM Rules#

Bucket policies or IAM permissions with conditional logic (e.g., Condition clauses) can cause intermittent denial if conditions aren’t always met. Examples include:

  • aws:CurrentTime conditions (e.g., access allowed only between 9 AM–5 PM).
  • aws:SourceIP conditions (e.g., allowing uploads only from your office IP, but failing when you work remotely).
  • s3:prefix conditions (e.g., allowing uploads to uploads/ but not temp/, if your app sometimes uses temp/).

3. Troubleshooting Steps: A Systematic Approach#

To resolve intermittent errors, follow this step-by-step process:

3.1 Reproduce the Issue Consistently#

First, create a minimal reproduction script to trigger the error reliably. For example, a Node.js script or browser-based HTML page that uploads 10–20 files in quick succession:

// Example: Node.js script to upload multiple files
const { S3Client, PutObjectCommand } = require("@aws-sdk/client-s3");
const fs = require("fs");
 
const s3Client = new S3Client({ region: "us-east-1" });
const bucketName = "your-bucket";
const files = Array.from({ length: 20 }, (_, i) => `file-${i}.txt`); // 20 test files
 
async function uploadFiles() {
  const uploadPromises = files.map((file) =>
    s3Client.send(
      new PutObjectCommand({
        Bucket: bucketName,
        Key: `uploads/${file}`,
        Body: fs.createReadStream(file),
      })
    )
  );
  await Promise.all(uploadPromises); // Upload all files concurrently
}
 
uploadFiles().catch(console.error);

Run this script repeatedly to confirm whether failures occur consistently under concurrency.

3.2 Enable Detailed Logging#

The AWS SDK for JavaScript (v3) allows logging request/response details, which is critical for debugging. Enable it by setting the AWS_LOG_LEVEL environment variable to debug:

# Node.js: Enable debug logging
AWS_LOG_LEVEL=debug node your-upload-script.js

In browsers, use the Network tab in DevTools to inspect S3 requests. Look for:

  • 403 Forbidden responses with AccessDenied in the XML body.
  • Preflight OPTIONS requests failing with 403 or 404.

3.3 Check Credential Validity and Refresh Logic#

If using temporary credentials, verify their expiry and refresh logic:

  • Log credential expiry times (e.g., credentials.expiration in the SDK).
  • Ensure your app refreshes credentials before they expire (e.g., using Cognito’s onTokenRefresh callback).

Example (Cognito):

// Ensure Cognito credentials are refreshed before expiry
cognitoUser.getSession((err, session) => {
  if (err) { /* handle error */ }
  const credentials = new AWS.CognitoIdentityCredentials({ IdentityId: session.getIdentityId(), Logins: { ... } });
  credentials.getPromise().then(() => {
    console.log("Credentials expire at:", credentials.expireTime); // Log expiry
  });
});

3.4 Verify CORS Configuration#

Check your bucket’s CORS policy for intermittent preflight failures:

  1. Go to the S3 Console → Your Bucket → Permissions → CORS.
  2. Ensure the policy allows your app’s origin, methods (PUT, POST), and headers (including Content-Type and any custom headers like x-amz-meta-*).

Valid CORS Policy Example:

[
  {
    "AllowedHeaders": ["*"], // Allow all headers (adjust for production)
    "AllowedMethods": ["PUT", "POST"],
    "AllowedOrigins": ["https://your-app.com"], // Restrict to your domain
    "MaxAge": 3000 // Cache preflight response for 3000 seconds
  }
]

In browsers, use DevTools’ Network tab to check preflight OPTIONS requests. A successful preflight returns 200 OK; failures return 403 or 404.

3.5 Inspect S3 Access Logs and CloudTrail#

S3 Access Logs and AWS CloudTrail provide granular details about denied requests:

  • S3 Access Logs: Enable them via the bucket’s Properties → Server access logging. Logs include the request timestamp, client IP, request headers, and error codes (e.g., AccessDenied).
  • CloudTrail: Look for PutObject events in CloudTrail. The errorCode field may reveal the root cause (e.g., ExpiredToken, InvalidSignature).

3.6 Test with Reduced Concurrency#

To rule out rate limiting or race conditions, test with sequential uploads instead of parallel:

// Upload files sequentially instead of with Promise.all
async function uploadSequentially(files) {
  for (const file of files) {
    await s3Client.send(new PutObjectCommand({ ... }));
    console.log(`Uploaded ${file}`);
  }
}

If failures disappear with sequential uploads, concurrency or rate limiting is likely the issue.

3.7 Validate SDK Version and Configuration#

Ensure you’re using the latest SDK version (v3 is recommended) and correct region:

# Check SDK version (Node.js)
npm list @aws-sdk/client-s3

Update with:

npm install @aws-sdk/client-s3@latest

Verify the SDK is configured for the bucket’s region:

const s3Client = new S3Client({ region: "eu-west-1" }); // Must match bucket's region

4. Solutions and Best Practices#

Once you’ve identified the root cause, apply these fixes:

4.1 Handle Temporary Credentials Gracefully#

  • Auto-Refresh Credentials: Use the SDK’s built-in refresh mechanisms (e.g., CognitoIdentityCredentials auto-refreshes when near expiry).
  • Extend Token Lifespan: For Cognito, increase the token expiry (up to 12 hours for server-side apps) via the Cognito User Pool settings.

4.2 Implement Retries with Exponential Backoff#

To handle rate limiting or transient network errors, use the SDK’s retry middleware with exponential backoff:

const { S3Client } = require("@aws-sdk/client-s3");
const { retryMiddleware } = require("@aws-sdk/util-retry");
 
const s3Client = new S3Client({
  region: "us-east-1",
  middlewareStack: (stack) =>
    stack.add(retryMiddleware({ maxRetries: 3, retryDelayOptions: { base: 1000 } })), // Retry 3x with 1s, 2s, 4s delays
});

4.3 Fix CORS Misconfigurations#

  • Allow all required headers (e.g., x-amz-acl, x-amz-meta-*) in your CORS policy.
  • Set MaxAge to a higher value (e.g., 3000 seconds) to cache preflight responses, reducing redundant requests.

4.4 Avoid Race Conditions with Asynchronous Uploads#

  • Limit Concurrency: Use a queue (e.g., p-queue) to restrict parallel uploads (e.g., 5 at a time).

Example with p-queue:

const { default: PQueue } = require("p-queue");
const queue = new PQueue({ concurrency: 5 }); // Max 5 concurrent uploads
 
async function uploadWithQueue(files) {
  const tasks = files.map((file) =>
    queue.add(() => s3Client.send(new PutObjectCommand({ ... })))
  );
  await Promise.all(tasks);
}

4.5 Simplify Conditional Policies#

If bucket policies or IAM rules use complex conditions, simplify them to avoid intermittent denial:

  • Remove time-based conditions unless strictly necessary.
  • Avoid IP-based conditions if users access the app from dynamic IPs (e.g., home vs. office).

5. Conclusion#

Intermittent S3 AccessDenied errors during multi-file uploads are rarely caused by static misconfigurations. Instead, they stem from dynamic factors like credential expiry, concurrency, CORS, and rate limiting. By systematically reproducing the issue, enabling detailed logging, and validating credentials/CORS, you can pinpoint the root cause.

Adopting best practices like credential auto-refresh, retries with backoff, and concurrency limits will ensure reliable uploads. With these steps, you’ll turn "sometimes broken" into "always working."

6. References#