Skip to content

dev: updates bulkexport script to batch exports#3125

Closed
gabestein wants to merge 22 commits intomainfrom
gs/batch-export
Closed

dev: updates bulkexport script to batch exports#3125
gabestein wants to merge 22 commits intomainfrom
gs/batch-export

Conversation

@gabestein
Copy link
Copy Markdown
Member

Issue(s) Resolved

  • Batches collection exports so they don't take up all the available db connections
  • Excludes pubs with no releases from exports by default

Test Plan

  • Export a large collection using the CLI tool
  • See that when exports are missing and need to be generate, it only tries to do ~5 at a time

Screenshots (if applicable)

Optional

Notes/Context/Gotchas

Supporting Docs

@gabestein gabestein requested a review from tefkah July 11, 2024 20:05
@pubpubBot pubpubBot temporarily deployed to pubpub-pipel-gs-batch-e-wormby July 11, 2024 20:19 Inactive
@pubpubBot pubpubBot temporarily deployed to pubpub-pipel-gs-batch-e-wormby July 11, 2024 21:22 Inactive
@gabestein gabestein changed the title updates bulkexport script to batch exports dev: updates bulkexport script to batch exports Jul 16, 2024
@pubpubBot pubpubBot temporarily deployed to pubpub-pipel-gs-batch-e-ldphaz September 10, 2024 22:37 Inactive
gabestein and others added 18 commits September 20, 2024 11:18
* redo landing page and remove some old routes

* get rid of landing page lint errors

* remove community creation links

* require superadmin to create community

* update links to kf.org

* Adjust jumbotron height

* update footer link

* fix tests

* better test fix

* ok actually do real tests fine gabe does tod now

* sigh spell it right

* I love tests

---------

Co-authored-by: Travis Rich <travis.s.rich@gmail.com>
…ocks (#3206)

* feat: add code attrs such as lang to statically rendered html code blocks

* fix: use data-lang instead of lang
* chore: upgrade heroku stack

* chore: try heroku 22

* chore: remove repo specifier

* chore: back to 24

* chore: deploy commit
* feat: add connection pooling

* fix: make sure DATABASE_READ_REPLICA_1 is set

* fix: actually set read replica env var correctly during test

* fix: always set the read_replica_db to the main db in tests

* fix: require second read replica
* fix: fix signup flow (ideally)

* fix: fix activity items

* fix: fix pubAttribution error

* fix: fix createThreadComment

* fix: useMaster for increment attemptCount
@pubpubBot pubpubBot temporarily deployed to pubpub-pipel-gs-batch-e-tugfgs April 1, 2025 19:57 Inactive
@pubpubBot pubpubBot temporarily deployed to pubpub-pipel-gs-batch-e-tugfgs April 1, 2025 20:25 Inactive
@isTravis
Copy link
Copy Markdown
Member

This PR is sufficiently out of date and the merge sufficiently complex that I think we'd have better luck approaching it fresh from the current codebase. I also think it's less of an issue now that the heroku pg database isn't in our way. Let's revisit if and when it comes up again!

@isTravis isTravis closed this Apr 16, 2026
@isTravis isTravis deleted the gs/batch-export branch April 17, 2026 02:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants