Skip to content

feat(transport): TUS upload for large attachments#1545

Merged
jpnurmi merged 54 commits intomasterfrom
jpnurmi/feat/tus
Apr 30, 2026
Merged

feat(transport): TUS upload for large attachments#1545
jpnurmi merged 54 commits intomasterfrom
jpnurmi/feat/tus

Conversation

@jpnurmi
Copy link
Copy Markdown
Collaborator

@jpnurmi jpnurmi commented Feb 27, 2026

  • Large attachments (>=100 MB) are stored as attachment-ref envelope items
  • Database layer stores large attachment files separately as db/cache/<uuid>-*
  • Transport layer streams each attachment-ref to the server via TUS before sending the envelope
  • Lays out the foundations with support for inproc and breakpad - native with daemon/IPC-related extra hoops comes separately in feat(native): TUS upload for large attachments #1684

Examples:

See also:

@jpnurmi jpnurmi force-pushed the jpnurmi/feat/tus branch 2 times, most recently from da2ce2d to 0946169 Compare March 2, 2026 09:32
@jpnurmi
Copy link
Copy Markdown
Collaborator Author

jpnurmi commented Mar 2, 2026

@sentry review

@jpnurmi
Copy link
Copy Markdown
Collaborator Author

jpnurmi commented Mar 2, 2026

@cursor review

Comment thread src/sentry_envelope.c Outdated
Comment thread src/sentry_database.c
Comment thread src/transports/sentry_http_transport.c
@jpnurmi
Copy link
Copy Markdown
Collaborator Author

jpnurmi commented Mar 2, 2026

@cursor review

Comment thread src/transports/sentry_http_transport.c
Comment thread src/sentry_database.c Outdated
Comment thread src/sentry_envelope.c Outdated
@jpnurmi
Copy link
Copy Markdown
Collaborator Author

jpnurmi commented Mar 2, 2026

@cursor review

Comment thread src/transports/sentry_http_transport.c Outdated
Comment thread src/transports/sentry_http_transport_curl.c
@jpnurmi
Copy link
Copy Markdown
Collaborator Author

jpnurmi commented Mar 3, 2026

@cursor review

Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

  • ✅ Fixed: copy_file_range fallback misses common filesystem errors
    • Expanded the copy_file_range fallback condition to also break on EXDEV, EOPNOTSUPP, and EINVAL so the read/write fallback handles these recoverable cases.

Create PR

Or push these changes by commenting:

@cursor push 76476611c4
Preview (76476611c4)
diff --git a/src/path/sentry_path_unix.c b/src/path/sentry_path_unix.c
--- a/src/path/sentry_path_unix.c
+++ b/src/path/sentry_path_unix.c
@@ -362,7 +362,8 @@
             goto done;
         } else if (errno == EAGAIN || errno == EINTR) {
             continue;
-        } else if (errno == ENOSYS) {
+        } else if (errno == ENOSYS || errno == EXDEV || errno == EOPNOTSUPP
+            || errno == EINVAL) {
             break;
         } else {
             rv = 1;
This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.

Comment thread src/path/sentry_path_unix.c
@jpnurmi jpnurmi force-pushed the jpnurmi/feat/tus branch from a69528d to 44d6ecc Compare March 3, 2026 09:27
@jpnurmi
Copy link
Copy Markdown
Collaborator Author

jpnurmi commented Mar 3, 2026

@cursor review

Comment thread src/sentry_database.c Outdated
@jpnurmi jpnurmi force-pushed the jpnurmi/feat/tus branch from 3bc1ee2 to 37004bf Compare March 5, 2026 08:03
@jpnurmi
Copy link
Copy Markdown
Collaborator Author

jpnurmi commented Mar 5, 2026

@cursor review

Comment thread src/sentry_envelope.c Outdated
@jpnurmi jpnurmi force-pushed the jpnurmi/feat/tus branch 3 times, most recently from eb0ead2 to 4ad89c0 Compare March 16, 2026 13:03
@jpnurmi jpnurmi force-pushed the jpnurmi/feat/http-retry branch from b133fd6 to 93db900 Compare March 17, 2026 16:15
@jpnurmi jpnurmi force-pushed the jpnurmi/feat/tus branch 3 times, most recently from 6d860cd to a07948d Compare March 19, 2026 16:33
Base automatically changed from jpnurmi/feat/http-retry to master March 31, 2026 19:23
@jpnurmi jpnurmi force-pushed the jpnurmi/feat/tus branch 4 times, most recently from c3f7b7f to bd32f9e Compare April 21, 2026 19:41
jpnurmi and others added 4 commits April 29, 2026 19:33
Separate retry enqueue handling from final cache_keep storage so client
reports are only preserved as retry state when an envelope is actually queued
for retry. Keep attachment-ref files only when the persisted envelope still
references them.

Prune any attachment-ref paths recovered from malformed raw envelopes before
dropping them to avoid leaving staged cache siblings behind.

Co-Authored-By: OpenAI Codex <noreply@openai.com>
Remove cache siblings by event id when a raw attachment-ref envelope cannot be
materialized. This avoids depending on partially parsed items, which may miss
attachment-ref paths from the corrupted part of the envelope.

Keep the UUID-based sibling cleanup in the cache layer alongside the existing
cache sibling removal code.

Co-Authored-By: OpenAI Codex <noreply@openai.com>
Keep cache_prune_siblings focused on the synchronous cleanup path. Enabling
cache_keep during init schedules transport cleanup in transport-enabled unit
runs, which can race with the test's direct cache setup.

Co-Authored-By: OpenAI Codex <noreply@openai.com>
Delete the destination path when staging an attachment-ref fails. Copy and
buffer-write failures can leave partial files behind, and those should not
remain as unreferenced cache siblings.

Co-Authored-By: OpenAI Codex <noreply@openai.com>
Comment thread src/transports/sentry_http_transport.c Outdated
jpnurmi and others added 2 commits April 30, 2026 07:19
Remove the global TUS availability flag. A failed TUS create request should
drop only the affected attachment-ref while still allowing later attachment
refs to try uploading.

This keeps upload errors local to each attachment-ref now that large uploads
are controlled by the explicit enable_large_attachments option.

Co-Authored-By: OpenAI Codex <noreply@openai.com>
Move TUS-specific unit tests from test_envelopes.c to test_tus.c so envelope
tests stay focused on envelope behavior.

Rename the TUS integration tests to shorter test_tus_* names and cover small
inline attachments in the upload happy path instead of a separate test. The
"no TUS at all" -case is already covered by the normal HTTP tests.

Co-Authored-By: OpenAI Codex <noreply@openai.com>
Comment thread src/sentry_attachment.c
jpnurmi and others added 2 commits April 30, 2026 07:42
Mention the Relay upload endpoint feature required for out-of-band large
attachments and link to the attachment placeholder docs.

Co-Authored-By: OpenAI Codex <noreply@openai.com>
jpnurmi and others added 2 commits April 30, 2026 10:32
When the shutdown timeout interrupts an in-flight TUS upload, keep the
attachment-ref cache siblings with the envelope dumped by the transport.
This lets the next startup retry the upload instead of losing the large
attachment.

Track timeout-caused send failures in curl and WinHTTP so normal upload
errors still drop the attachment-ref and send the envelope without it.

Co-Authored-By: OpenAI Codex <noreply@openai.com>
Comment thread src/transports/sentry_http_transport.c
jpnurmi and others added 2 commits April 30, 2026 11:36
Treat attachment-ref resolution failures after a successful TUS upload as a
transport error instead of discarding the reference. This keeps the event and
uploaded attachment reference together for retry or cache handling.

Co-Authored-By: OpenAI Codex <noreply@openai.com>
Comment thread src/transports/sentry_http_transport.c
Comment thread src/sentry_envelope.c
Comment thread src/sentry_core.c
Comment thread src/transports/sentry_http_transport.c
Comment thread src/transports/sentry_http_transport.c Outdated
Fall back to the existing retry/cache/discard handling when persisting a
TUS shutdown envelope fails. This restores the saved client report and
keeps discard accounting consistent with other network errors.

Co-Authored-By: OpenAI Codex <noreply@openai.com>
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 730b3a2. Configure here.

Comment thread src/sentry_database.c
Comment thread src/sentry_envelope.c
@jpnurmi jpnurmi merged commit cbe330b into master Apr 30, 2026
59 of 60 checks passed
@jpnurmi jpnurmi deleted the jpnurmi/feat/tus branch April 30, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants