HDDS-10611. Design document for MPU GC Optimization

[spacemonkd]

What changes were proposed in this pull request?

HDDS-10611. Design document for MPU GC Optimization

Please describe your PR in detail:
This PR adds the design doc for optimizing the OM GC pressure by the MPU file handling

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10611

How was this patch tested?

N/A

[ivandika3]

Thanks @devabhishekpal for the design on this long overdue issue. I am +1 on the overall direction. Left some comments.

[spacemonkd]

Thanks for the exhaustive review and inputs @ivandika3.
I updated the document with the new details, please do let me know in case I am missing something in the understanding and also if something else could be improved.

FYI, I have a sample/PoC patch created if anybody wants to check the changes.
master...devabhishekpal:ozone:HDDS-10611

[errose28]

Thanks for the design @devabhishekpal @rakeshadr. Overall LGTM, just a few things we can clarify in the doc.

[spacemonkd]

@ivandika3 @errose28 could you take another look at the current design doc?

[ivandika3]

Thanks for the update and the effort. LGTM +1.

Since we are introducing a flattened schema, we need to be mindful that RocksDB tombstones issue might happen if there are a lot of MPU aborts with large number of parts. So it might be important to put this new MPU metadata table as the list of compacted table (OZONE_OM_COMPACTION_SERVICE_COLUMNFAMILIES_DEFAULT).

[errose28]

Looks good, thanks @devabhishekpal for the write-up and iterative improvements. The only open comment I see is this one on using deprecated labels for proto fields but we can make that decision in the implementation.

[spacemonkd]

Hi Ethan, thanks for the approval. I had forgot about this. However we have decided to move on with approach 1 which plays better with backward compatibility concerns.

The related protobuf changes for this has been merged already and there we have marked the field as deprecate where applicable.
Ref #9867 which introduces the protobuf changes.

[spacemonkd]

Thanks for the reviews and inputs @errose28 @ivandika3 @ChenSammi @jojochuang.
Merging this design doc.

[mango-li]

If updating the multipartInfoTable is still necessary here, can we retain some information from the partKeyInfoList, such as the part name and part number, but remove the partKeyInfo? This way, future requests can avoid scanning the multipartPartTable and only need to query a single key.