Most tasks swallow exceptions

[llucax]

What happened?

When there is an unhandled exception inside a task spawn by the SDK, the error is silently swallowed, leading to obscure, hard-to-debug bugs.

What did you expect instead?

Unhandled exceptions are at least logged or the whole Python process crashes.

Affected part(s)

Core components (data structures, etc.) (part:core)

Extra information

I discovered this while working at #806. I added a sanity check which failed inside a task that was sending messages to a channel, so no messages was sent, and other task waiting for messages just got stuck, leaving no clues about where the problem might be. This makes the problem really hard to debug.

This issue is actually a problem across all areas of the SDK and projects in general.

A way to cope with it in the SDK would be to extend BackgroundService to provide a create_task() method that automatically adds the task to the task list and then also adds a done callback where we can either log the unhandled exception, or just raise a SystemExit exception to exit the program. We could even give the user the option to decide how to handle unhandled exceptions by either passing a callback or letting them override the default callback as a method of the instance. This callback should also remove the task from the tasks list, something users need to do manually at the moment.

Related issues

The solution to this issue needs to have in mind the following related issues:

• Add a safety mechanism to BackgroundService to notify about crashed tasks frequenz-core-python#9
• Replace run_forever implementations with a generic one in _internal #906
• Improve the Actor interface to handling internal tasks #819
• Make actor automatic restarting more resilient #632