missing out on the benefit of queue-based ingestion

[twrichards]

grid-tools/migrate.py

Line 22 in ea88fc2

def iterate_images(process_image):

this approach of using the image-loader HTTP endpoints misses out on the benefits of queue-based ingestion and could fall over with giant images (and run sub-optimally for smaller images). Instead, I would suggest...

1. using the API to compile a list of media IDs (see probably best to use offset rather than since for paging in migration #1 though!), or rather pre-produce the metadata JSON payloads as files, with the mediaId as the filename
2. give GetObject permission to the source image bucket, iterate the files from Step 1, performing an S3 copy to the `destination' grid's ingestion queue bucket - blitz through those as fast as possible (we should add support for uploadTime S3 metadata on the queue bucket, such that it gets written to the file which ends up in the image bucket after ingestion)
3. wait for all the images to be ingested, then perform the metadata updates similar to the current script using the JSON files from Step1

[tonytw1]

These considerations where in mind when choosing to POST the image date to the API synchronously:

• It's black box with and only needs Grid API keys.
• It can use the upload time overrides on the POST endpoint to preserve upload times.
• It completes the setting of the metadata for the current image before moving onto the next one; images in the destination Grid will be in a finished state as they appear.

The ingest queue was discounted because:

• It was assumed AWS access would be needed to add images to the queue. Didn't want the user to have to know any of the AWS internals.
• The upload time overrides weren't available via the queue.

After speaking IRL, these points where brought to my attention which could make ingesting a really nice solution:

• There is a prepare end point in the Grid API which might issue presigned S3 URLs. This might be a way for a user with only a Grid API key to push images onto the ingest queue.
• We could consider implementing metadata overrides on the ingest queue using S3 metadata fields.
• We could consider putting the entire Grid metadata object onto the S3 metadata.

[twrichards]

• There is a prepare end point in the Grid API which might issue presigned S3 URLs. This might be a way for a user with only a Grid API key to push images onto the ingest queue.

correct (and time permitting I will try to add support for setting upload time via this prepare endpoint, using S3 metadata on the object in the queue bucket)

[twrichards]

• We could consider putting the entire Grid metadata object onto the S3 metadata.

we'd need to check what the limits are and we'd also need to think whether we want image-loader dispatching the metadata message(s) to thrall

would be elegant to do it one transaction per image though