some difficulties using the DeDRM standalone stuff

[cameron-simpson]

I've been running this successfully outside Calibre for some years. This is the module in my cs.ebooks package which hooks into the DeDRM stuff:
https://github.com/cameron-simpson/css/blob/ebooks/lib/python/cs/ebooks/dedrm.py#L228
The interesting bit is the DeDRMWrapper class.

Some of the elaborateness is because I'm deliberately not modifying any DeDRM code.

Currently it works against a source checkout of the git repo (or did as of DeDRM 7.2.1), and I'm updating it to also be able to find the DeDRM_plugin.zip in the local Calibre install and run off that, and things have started breaking (possibly due to me foolishly updating from 7.2.1, but I need to work with current stuff anyway at some point).

Currently I've got a stopper in standalone/__init__.py which contains this code:
https://github.com/noDRM/DeDRM_tools/blame/3373d938749117989dfd0a42cc947bf338d3ffda/DeDRM_plugin/standalone/__init__.py#L75

This imports from standalone/__init__.py itself, because it pushes the standalone directory onto the front of sys.path. Needless to say, the import of these symbols fails.

I can pursue this with a monkey patch (ugh!), but do you want to pursue fixes at your end?

[cameron-simpson]

Ok, I've a workaround.

When I go to load the DeDRM modules I temporarily modify sys.path like this:

    with stackattrs(
        sys,
        path=[
            # shims first
            self.shimlib_dirpath,
            # then the dedrm package itself, since it directly imports
            # with absolute names like "kindlekeys"
            joinpath(self.shimlib_dirpath, self.DEDRM_PACKAGE_NAME),
        ] + sys.path + [
            # dedrm/standalone/__init__.py self inserts its directory
            # at the start of the path if it isn't already somewhere
            # in sys.path (this is bad)
            # https://github.com/noDRM/DeDRM_tools/issues/653
            joinpath(self.shimlib_dirpath, self.DEDRM_PACKAGE_NAME,
                     'standalone'),
        ],
    ):

This inserts the directory containing some small shim modules I've made to interface with the DeDRM stuff, then the directory containing the DeDRM modules themselves, since they import adjacent modules using absolute names like kindlekeys.

Then it appends the standalone directory to the end of sys.path so that standalone.__init__.py does not decide to prepend its directory to the front, which is what's breaking the import of the DeDRM version etc symbols.

[noDRM]

I've changed a bit of stuff around compared to 7.2.1 when i started to try and implement my own standalone version of the plugin (which is not ready for use yet). With the current version, both running the plugin from Calibre and running the plugin standalone seems to be working (or at least, worked when I last tested it).

Not sure how this all works when including the plugin from another python project, I don't think I ever tested that. Is there a particular change I can make to the plugin to help your use-case while not breaking the Calibre plugin nor the direct standalone execution? Would moving the standalone directory from the beginning of the path variable to the end fix the issue?

I can take a look at that and see if I can implement that without breaking any existing functionality.

[cameron-simpson]

Well I certainly don't want to break the plugin side.

Moving the standalone directory to the end if missing would help. That would prevent it preemptying other things.

If I had a free hand and you wanted my input or collaboration, I'd like the DeDRM stuff to look like a package inside the zip file so that it wasn't importing its components as absolute names. Ideally it would use relative imports. A from __future__ import absolute_imports would make that behaviour predictable and work on older Pythons (back to... somewhere I can look up).

All of this with the caveat that I don't know what degree of backward compatibility the plugin strives to support. I know there even used to be some Python-2-isms in the code back in 7.2.1 when I last read it in detail.

If it were me, my long term goal would be:

• make the .py files all be in a package inside the plugin, eg a dedrm.* package
• including the standalone stuff, so that someone outside Calibre could install the package and then import dedrm and so forth in some fashion

My personal short term goal is to make my own code work against the plugin without modifying the plugin. But it would be good to make that easier to do, for me and for others.

It would be good to pin down what backwards compatibility stuff is required (i.e. what constructs we must avoid inside the plugin code). And to consider whether there's a point where the tool splits a little into very-backwards-compatible branch and a modern-with-easy-external-standalone branch.

[cameron-simpson]

I need to investigate how feasible it is to make a calibre plugin contain a nice distinctly named package. By writing something trite myself first. I think it's feasible, and if it is it becomes tractable to make a proper package, and in the plugin zip have the package and also the necessary shim to hook into it. Which would leave you a freer hand for the standalone mode.

[noDRM]

Right now, the plugin is still supporting Python 2.7 so people can still use it on Calibre 4 and below, so ideally any packaging changes would also support that. from __future__ import absolute_imports looks like it's been added in Python 2.5 so that should be fine. There's discussions (in #662) about dropping Python2 support, but for now I'd prefer if it was still supported.

That said, I'm only still supporting Python 2.7 for people stuck on older Calibre versions. As long as it doesn't affect loading the plugin in Calibre with Python2, any additional code only used / loaded only for the standalone version outside of calibre might as well drop support for the old versions and require 3.8+ or even 3.10+ - if that's possible, not sure how exactly python module loading works.

As for making the DeDRM plugin a normal import-able python module - it'd be great if that was possible somehow. The "standalone" code (which I started to work on and then never had time to finish) was intended so more people could use this plugin even without calibre. If this can somehow be expanded so people could even use this as a normal python module in their applications, that'd be great.

Unfortunately, due to some oddities in how Calibre uses imports / loads modules, even getting the standalone version to work was tricky and required messing with the paths. As far as I know, Calibre loads the module into its own namespace and you have to use that namespace to access other plugin files in some cases - which is why there's even that mess with the init.py, main.py, version.py and calibre_compat_code.py - because I wasn't able to get it to work otherwise.

It's definitely a hacky solution and if someone with more Python knowledge finds a way to clean up this mess so that all three code paths (calibre plugin, standalone execution, importing as a python module) still work, I'd definitely be interested in adding the required changes for that.