Find in batches

[maxkadel]

For large repositories, allow objects to be found in batches, so that all objects don't have to be stored in memory in order to perform a process on all of them.

Prior art: Figgy's memory_efficient_all_query.

See also Slack discussion

[tpendragon]

Oh interesting, someone just improved this in valkyrie-sequel, I wonder if a similar more generalized strategy is useful here? https://github.com/samvera-labs/valkyrie-sequel/pull/51/files

I wonder how we could test it. 🤔

[maxkadel]

I was thinking of just passing it on to the ActiveRecord find_in_batches method for Postgres, since it already handles a lot of the transaction / cursor logic.

I'll push up what I have for Postgres - it won't be passing since I haven't finished the methods for the other query services.

Although, if we're trying to move away from Rails / ActiveRecord... 🤔

[dchandekstark]

Note that find_each uses find_in_batches - https://api.rubyonrails.org/classes/ActiveRecord/Batches.html#method-i-find_each.