Bug: jupyter server management is very brittle

[jlewi]

The initial management of jupyter servers in
branch: dev/jlewi/jupyter
commit: c1b5319

Is very brittle. The issue is we rely on token auth between the runme server and the jupyter server. The runme server then maintains a cache of the tokens/jupyter server configuration e.g. port. When starting jupyter server we need to write a .json file to the .runme-agent/jupyter directory.

There is all kinds of brittleness in this initial design. A lot of the issues go back to the double hop
runme -> jupyter_server -> ipykernel.

This was discussed in the original design in #63.

Some options are

1. Get rid of jupyter_server and reimplement websockets -> zmq inside runme.
2. Get rid of runme and talk to jupyter_server directly

Could we use OIDC with Jupyter server?
https://jupyter-server.readthedocs.io/en/latest/operators/security.html#authentication-and-authorization

I think we'd need an IdentityProvider/Authorizer plugin to validate the JWT and authorize users based on it.

[jlewi]

I couldn't find an off the shelf JWT provider.

https://github.com/greenpau/jupyter-caddy-security
Was the closest I could come to prior art.

I lean towards the getting rid of jupyter_server and reimplementing ws -> zmq in golang. Its more work upfront but I think in the long running not managing two servers will be a better UX. If nothing else it means we only need to deal with a single port/process when running in a remote server