Refactor task-to-call conversion

[jacob720]

From discussion with @DominicOram about making the queue service as generic as possible:

The queue service currently is set up for converting experiment definitions into blueapi calls. However, the way this conversion works will need to change depending on what else is in the queue. For example, an experiment may require a calibration scan to be ran before it. However, without knowledge of what else is in the queue, the same calibration scan may be queued up multiple times for different experiments that could actually share the same calibration data.

To solve this, the conversion step should actually be from an entire queue of tasks (and maybe history) to a queue of blueapi calls. We now have two queues:

• Task Queue - should be manipulatable by a beamline scientist
• Call Queue - only able to be mutated/created by the conversion function - and it's this queue that provides to a get_next_thing endpoint called by the worker (or external service Pull the queue worker into a seperate service #17)

Every time anything changes in the Task Queue (things added, deleted moved), the conversion will need to be re-calculated. Any time get_next_thing is called, the conversion should be re-calculated, as current conditions on the beamline may inform the conversion function.

Acceptance Criteria

• Converter functions take the entire queue and history as inputs, and outputs a queue of blueapi calls
• The conversion is ran whenever anything about the queue changes

[jacob720]

Questions this raises:

• To keep the queues in sync, a call will need a reference to its parent task, and probably vice versa.
• Should calls have their own unique ID? If so, it will be regenerated every time the conversion is ran? No, the items in the Call Queue are not persistent and could change constantly due to the conversion
• Should the Call Queue affect the Task Queue? For example, if the conversions means that the Call Queue order does not reflect the Task Queue, should the task queue order change so that it is a better representation of the order of events? This becomes complicated if the conversion produces interleaved events. No
• What are the statuses of the objects in each queue? Does the task status depend on the statuses of the calls it has produced? Yes, for now, the status of the task depends on the status of its child blueapi calls
• Can you move/cancel a task if the call it has produced is in progress? If the task produces multiple calls and some are complete/in progress, can you move or delete the task? No, once any child call of a task has started, it is in progress and can't be moved
• Presumably a task should leave the queue once all of its child tasks have been completed. Yes
• For tasks that produce multiple calls, how can we prevent the conversion creating a duplicate call once a call is complete? The conversion needs to take the call history and be aware of that? This makes it very easy to write bad converters. For now, lock a task and its children calls once one of them starts