SigV4 authenticated uploads fail with SignatureDoesNotMatch on Cloudflare Workers

[diegoripley]

Here's a more general GitHub issue:

What I'm trying to do

I'm trying to use rclone to upload files to a multistore proxy running on Cloudflare Workers. Anonymous reads work fine, but authenticated writes (PUT) and authenticated list operations (GET with credentials) all fail with 403 SignatureDoesNotMatch.

Has anyone gotten authenticated uploads working with the Cloudflare Workers deployment?

My setup

rclone config:

[dataforcanada-s3]
type = s3
provider = Other
endpoint = https://s3.dataforcanada.org
no_check_bucket = true
access_key_id = <my-access-key>
secret_access_key = <my-secret-key>
env_auth = false

Upload command:

rclone copy . dataforcanada-s3:dataforcanada/d4c-datapkg-orthoimagery/

Wrangler config (sanitized):

compatibility_date = "2024-11-11"
main = "build/worker/shim.mjs"
name = "dataforcanada-s3"

[build]
command = "cargo install worker-build && worker-build --release"

[vars]
VIRTUAL_HOST_DOMAIN = "s3.dataforcanada.org"

[[routes]]
pattern = "s3.dataforcanada.org"
custom_domain = true

[vars.PROXY_CONFIG]

[[vars.PROXY_CONFIG.buckets]]
allowed_roles = []
anonymous_access = true
backend_type = "s3"
backend_prefix = "dataforcanada/"
name = "dataforcanada"

[vars.PROXY_CONFIG.buckets.backend_options]
bucket_name = "us-west-2.opendata.source.coop"
endpoint = "https://s3.us-west-2.amazonaws.com"
region = "us-west-2"
access_key_id = "<backend-key>"
secret_access_key = "<backend-secret>"
skip_signature = "true"

[[vars.PROXY_CONFIG.credentials]]
access_key_id = "<proxy-access-key>"
created_at = "2025-01-01T00:00:00Z"
enabled = true
principal_name = "dataforcanada-admin"
secret_access_key = "<proxy-secret-key>"

[[vars.PROXY_CONFIG.credentials.allowed_scopes]]
actions = [
  "get_object",
  "head_object",
  "put_object",
  "delete_object",
  "list_bucket",
  "create_multipart_upload",
  "upload_part",
  "complete_multipart_upload",
  "abort_multipart_upload",
]
bucket = "dataforcanada"
prefixes = []

[placement]
mode = "smart"

[observability]
enabled = true

What I see

Every authenticated request fails with a SigV4 signature mismatch. From wrangler tail:

[WARN] multistore::auth::sigv4: SigV4 signature mismatch { access_key_id=<my-key>, region=us-east-1 }
[WARN] multistore::proxy: request failed { error=signature mismatch, status=403, s3_code=SignatureDoesNotMatch }

I can provide more detailed logs if you need them.

Question

Has anyone successfully used authenticated uploads (PutObject) with the Cloudflare Workers deployment?

[alukach]

So do you think this signature issue is from the upstream S3 server or from the multistore server?

[alukach]

In 0e8b416, I've added integration tests that test the upload (with a Minio backend):

multistore/examples/cf-workers/wrangler.integration.toml

Lines 46 to 61 in 0e8b416

	[[vars.PROXY_CONFIG.credentials]]
	access_key_id = "AKTEST000000000001"
	created_at = "2024-01-01T00:00:00Z"
	enabled = true
	principal_name = "integration-test-user"
	secret_access_key = "testSecretKey00000000000000000001"
	[[vars.PROXY_CONFIG.credentials.allowed_scopes]]
	actions = ["get_object", "head_object", "put_object", "delete_object", "list_bucket"]
	bucket = "public-data"
	prefixes = []
	[[vars.PROXY_CONFIG.credentials.allowed_scopes]]
	actions = ["get_object", "head_object", "put_object", "delete_object", "list_bucket"]
	bucket = "private-uploads"
	prefixes = []

[diegoripley]

I updated my config to match the integration tests, removing the multipart upload actions:

Before:

[[vars.PROXY_CONFIG.credentials.allowed_scopes]]
actions = [
  "get_object",
  "head_object",
  "put_object",
  "delete_object",
  "list_bucket",
  "create_multipart_upload",
  "upload_part",
  "complete_multipart_upload",
  "abort_multipart_upload",
]
bucket = "dataforcanada"
prefixes = []

After:

[[vars.PROXY_CONFIG.credentials.allowed_scopes]]
actions = ["get_object", "head_object", "put_object", "delete_object", "list_bucket"]
bucket = "dataforcanada"
prefixes = []

The [[vars.PROXY_CONFIG.credentials]] section is identical to yours.

I also reproduced the issue using the AWS CLI (not just rclone):

This is downloading from my multistore deployment at https://s3.dataforcanada.org/dataforcanada (which points to Source Cooperative) with AWS credentials removed from the config.

[diegoripley]

Once this is working, I'll route ingestion through an HTTP ingestor Cloudflare worker — which fetches the file at the edge, minimizing egress from the origin — then forward it to my multistore deployment, positioned to minimize latency to Source Cooperative.

Right now I just have the HTTP ingestor uploading to Source Cooperative. It works, but I think it can work better with multistore. So far I have ingested this much, but it's a 100TB dataset so I have to be more strategic.