[Proposal] A cleaner and clearer files model and interface

[alexdutton]

Motivation

The current implementation of invenio-files-rest is in places confusing to follow and doesn't provide an easy-to-use Python API or allow for easy extensibility to support multiple storage backends in one system.

See also related discussion on inveniosoftware/invenio-files-rest#255.

Summary

Proposed changes

• Storage backends are registered using entry points, and a(n overridable) name→class mapping added to config
• FileInstance.storage_class is migrated from String(1) to a larger string, and used to point to the storage backend entry point name used for the file. It is currently unused.
• A new storage factory is created that uses FileInstance.storage_class to mediate access to stored files based on .storage_class
• The new storage factory is the default for new cookiecutters, but the old pyfs storage factory remains with the old behaviour and a DeprecationWarning, to provide a migration path
• The new storage factory can be overridden to use another way to determine storage class for both existing files and new files. I expect to make this a class-based factory, with obviously overrideable methods.
• An alembic migration that ALTERs FileInstance.storage_class to extend its length and populates it with according to some (yet to be determined) heuristic
• FileInstance.storage(**kwargs) has its **kwargs deprecated. Possibly deprecate in favour of a (cached) file property, for a cleaner interface. New interface is e.g. with file_instance.file.open() as f:
• Update documentation to reflect changes
• Updates to invenio-s3 to use the new storage framework implementation

Resources

Cottage Labs is offering 2 days a week of developer effort that could be spent on this. I haven't yet estimated the amount of effort required.