Skip to content

feat: remove old archived email files from S3#4830

Merged
joybytes merged 3 commits into
mainfrom
cleanup/archived-template-email-files-s3
May 11, 2026
Merged

feat: remove old archived email files from S3#4830
joybytes merged 3 commits into
mainfrom
cleanup/archived-template-email-files-s3

Conversation

@joybytes
Copy link
Copy Markdown
Contributor

@joybytes joybytes commented Apr 24, 2026

Summary

Added a daily task to remove archived template email files from S3. By default, it cleans a 1-day slice of files that are at least 14 days old. You can pass archived_from to backfill older files.

Testing

Without argument
Running the task without archived_from uses the default cleanup window:

  • archived_to = now - 14 days (retention cutoff)
  • archived_from = archived_to - 1 day So it only targets files archived in that 1-day window (between 15 and 14 days old), not all files older than 14 days.
docker exec -it notify-api-celery \
  celery -A run_celery.notify_celery call remove-archived-template-email-files-from-s3 \
  --queue=periodic-tasks

With argument
Providing archived_from widens/narrows the start of the window for backfill:

  • lower bound becomes your provided archived_from
  • upper bound is still archived_to = now - 14 days This means retention is still enforced + files newer than 14 days are never included.
docker exec -it notify-api-celery \
  celery -A run_celery.notify_celery call remove-archived-template-email-files-from-s3 \
  --queue=periodic-tasks \
  --kwargs='{"archived_after":"2026-04-17"}'

Ticket

Automatically remove stale email files from S3

@joybytes joybytes force-pushed the cleanup/archived-template-email-files-s3 branch 7 times, most recently from 5271170 to 1c9f19c Compare April 28, 2026 14:57
@joybytes joybytes marked this pull request as ready for review April 28, 2026 14:58
Copy link
Copy Markdown
Contributor

@CrystalPea CrystalPea left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, good work on this, well done!

I left some comments. This is not the full review yet - Github seems to be playing up and losing some of my comments, so I want to submit to see what came through and avoid repeating myself. I will finish the review in a second swipe now.

Comment thread app/celery/nightly_tasks.py Outdated
Comment thread app/celery/nightly_tasks.py Outdated
Comment thread app/dao/template_email_files_dao.py Outdated
Comment thread app/dao/template_email_files_dao.py Outdated
Comment thread tests/app/dao/test_template_email_files_dao.py Outdated
Comment thread tests/app/dao/test_template_email_files_dao.py Outdated
Comment thread tests/app/dao/test_template_email_files_dao.py Outdated
Comment thread app/constants.py Outdated
Comment thread app/celery/nightly_tasks.py Outdated
Comment thread app/celery/nightly_tasks.py Outdated
@CrystalPea
Copy link
Copy Markdown
Contributor

I reviewed the rest of the PR and I have no more comments 🙌🏼

@joybytes joybytes marked this pull request as draft May 1, 2026 15:47
@joybytes joybytes force-pushed the cleanup/archived-template-email-files-s3 branch 3 times, most recently from 8648d63 to 46cdd68 Compare May 5, 2026 17:33
@joybytes joybytes force-pushed the cleanup/archived-template-email-files-s3 branch from 46cdd68 to 553628a Compare May 5, 2026 17:42
@joybytes
Copy link
Copy Markdown
Contributor Author

joybytes commented May 5, 2026

@CrystalPea could you give a final look to the implementation of dao_get_archived_template_email_files_older_than please? 🙏

I did some refactoring, and I am now using older_than and .scalar()

I noticed that the other keyset/cursor functions in our repo seems to use this pattern, so I tried to follow the same
(eg.dao_get_paginated_inbound_sms_for_service_for_public_api, get_notifications_for_service)

I also decided to use PAGE_SIZE config variables, which is what inbound notifications seem to use, so I just decided to follow the same limit for sizing

@joybytes joybytes marked this pull request as ready for review May 5, 2026 18:16
@joybytes joybytes merged commit 405721f into main May 11, 2026
10 checks passed
@joybytes joybytes deleted the cleanup/archived-template-email-files-s3 branch May 11, 2026 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants