Stepping up the LECO formalism to the next level

[Ashwolaa]

LECO and ZMQ: decoupling hardware from DAQ modules

The LECO protocol makes managing communication between the different entities in an acquisition software very interesting. The Actor/Director/Coordinator model, once understood, is particularly powerful, and some extensions of these concepts could be very fruitful.

I have been thinking about improving the communication flow between the different components inside and outside PyMoDAQ. A powerful tool at the heart of PyLECO is ZMQ. I have started using it in my own software development and it allows easy asynchronous
data distribution over a network.

A promising direction is to entirely decouple the hardware layer from the DAQ modules and instead fully associate it with the Actor. The DAQs would be limited to the role of directors that query or change data. One could say that this is already the case — a DAQ has to implement specific methods to get information from the hardware. However, the coupling between the DAQ and the hardware is currently much tighter than it needs to be.

Let me first introduce the notion of Observable and Variable.

Observable vs Variable

Every quantity a hardware instrument exposes falls into one of two categories:

• Observable — something we can measure. Read-only from the outside. A DAQ_Viewer director is built on observables.
• Variable — something we can change. Read-write. A DAQ_Move director requires at least one variable. Every Variable is also an Observable, as we usually need to read it back — for example, following a stage moving to verify it reached its target position.

Unified hardware interface

At the hardware level, all instrument interaction can thus be reduced to two operations:

• query_data → get an Observable
• change_to → change a Variable

The point of this semantic distinction is that by exposing read/write methods targeting specific elements of the hardware, it becomes straightforward to derive a DAQ module from an instrument driver. A reading method translates to a DAQ_Viewer director asking the Actor to update a value, while a writing method gives rise to a DAQ_Move director.
I believe this is also a bit closer to the original approach of PyLECO in which the Actor and the hardware are linked within one another.

Three key differences from the current approach

1. No master/slave conflicts

In the current architecture, the master/slave distinction can lead to read/write conflicts when multiple modules share access to hardware. In the proposed model there is exactly one Actor per hardware instrument, and it is the only entity that ever touches the
hardware directly. DAQ modules connect to the Actor through the LECO protocol and send RPC commands to make it act as they wish. Conflicts between concurrent hardware accesses disappear by construction.

2. Data channel via ZMQ Proxy

Instead of routing measurement data back and forth between the Actor and the Director through the Coordinator — which is designed for control flow, not bulk data — a dedicated Data Proxy (ZMQ XPUB/XSUB) dispatches published data from the Actor to any number of
subscribers over the network. This separates two fundamentally different communication patterns:

• Control flow (commands, queries, settings): small JSON payloads, bidirectional, through the Coordinator.
• Data flow (measurements, detector frames, positions): potentially large binary payloads, one-way broadcast, through the Data Proxy.

This separation also enables a new concept: the Spectator (see below).

3. Modularity: one Actor, many Directors simultaneously

Because the Actor is fully decoupled from any specific DAQ module, multiple directors can connect to the same Actor at the same time. A scan director, a live viewer, and a data logger can all subscribe to the same camera Actor simultaneously, each receiving every published frame independently. No special coordination between them is required —the ZMQ PUB/SUB pattern handles fan-out natively.

This also means that an Actor can be shared across different PyMoDAQ instances running on different machines, or even across different software frameworks that speak LECO similarly to what is currently implemented with the DAQ_move and DAQ_viewer

Conceptual model

   ┌─ Machine A (hardware side) ──────────────────────────────────┐
   │                                                              │
   │   LECO Coordinator ◄──────────────── (links to other         │
   │         |                             Coordinators)          │
   │   Data Proxy (ZMQ XPUB/XSUB)                                 │
   │         |                                                    │
   │   PymodaqActor                                               │
   │     ├── RPC methods ──────────────► Coordinator              │
   │     └── DataPublisher ────────────► Data Proxy ──────────┐   │
   └──────────────────────────────────────────────────────────┼───┘
                                                              │
   ┌─ Machine B (client side) ────────────────────────────────┼──┐
   │                                                          │  │
   │   Director(s)          ─── RPC via Coordinator ───────►  │  │
   │   (DAQ_Move, DAQ_Viewer)                                 │  │
   │                                                          │  │
   │   Spectator(s)         ─── ZMQ SUB ◄─────────────────────┘  │
   │   (subscribe-only)                                          │
   └─────────────────────────────────────────────────────────────┘

The Spectator concept

A Spectator is a lightweight subscriber that connects only to the Data Proxy. Unlike a Director, it has no RPC channel to the Actor and cannot issue commands. It simply subscribes to one or more Actor topics and receives every published frame.

Spectators are useful for:

• Live monitoring: a separate display process that shows the current state of an instrument without interfering with an ongoing acquisition.
• Data logging: a background process that records every published frame to disk, independently of what the Directors are doing.
• Cross-instrument analysis: a Spectator can subscribe to several Actors simultaneously and correlate their data streams in real time — for example, synchronising a position readout with a detector frame.

Because the ZMQ PUB/SUB pattern is purely additive — adding a new subscriber never affects the publisher or existing subscribers — Spectators are completely transparent to both the Actor and the Directors.

@seb5g @malik-irain @rgeneaux @quantumm @BenediktBurger

[BenediktBurger]

Actually, PyLECO already contains the infrastructure for that PUB SUB communication in its "data protocol" (vs. the "control protocol" which exchanges messages back and forth).
The PyLECO Listeners already listen to both, incoming control (via Coordinator) and data messages (via SUB).
For a spectator, you just need to use that Listener.

I like the decoupling, which was the goal of LECO, as my own measurement software got hickups with concurrent read/writes, leading me to write LECO.

[seb5g]

@BenediktBurger: To your experience, what's the overhead of going through leco compared to communicating directly as it is now?

This proposal also reminds me of YAQ which I don't like because you always have your instruments hanging around and no direct control. It goes in the direction of other framework like tango, bluesky and we have to be carefull of what/who we target with pymodaq. Can we do a visio conf on the subject to exchange our views? @Ashwolaa do you want to chair it?

[BenediktBurger]

To your experience, what's the overhead of going through leco compared to communicating directly as it is now?

Performance-wise I never noticed anything. But I did not send large files in a short time (e.g. camera feed) through LECO.
Once I tried (see #405) to add LECO publishing of data parallel to the normal inner workings (Qt Signals). Somehow the window controlling the camera froze but the window doing remote control got all the images etc. I was not able to diagnose this (probably multi-threading, qt-signal etc.) issue as I did not have the capacity to investigate it deeply (I never used PyMoDAQ).

Keep in mind, though, that LECO requires plain bytes, so you might have to turn the Python object your camera driver gives you into a bytes object and later into a Python object again. If you get a bytes (like) object, you can just push it through the line. Qt Signals can handle Python objects.

This proposal also reminds me of YAQ which I don't like because you always have your instruments hanging around and no direct control.

Regarding hanging around: Yes, they have to be somewhere. I used a "task manager" (called starter) which could start/stop these instrument drivers on demand, but keep them all together. The advantage, however, is that you have them available to several PyMoDAQ sessions and other programs flexibly. And typically, they just sit around waiting.

Regarding no direct control: You already have the same: you just communicate with the hardware in another thread via signals, so you could replace the signals with LECO messages ("leco signals").

My thoughts:

Pros

• you separate the GUI/control/data collection from the hardware access (separation of concern)
• it does not matter, whether GUI and instrument live in the same process and remote access (other process, other computer) is equally as local access
• Removes one layer of threading of a module (it took me quite some time to understand these different layers of hardware, plugin and main module class)

Cons

• you have to manage the instrument drivers (or offer something to manage it), which allows to start them easily and to stop them
• messages from the wrong instrument could arrive at the wrong module. Signal connections are more "direct"
• You have only binary transport (only bytes) -> might be a performance issue (I don't know)

[Ashwolaa]

Actually, PyLECO already contains the infrastructure for that PUB SUB communication in its "data protocol" (vs. the "control protocol" which exchanges messages back and forth). The PyLECO Listeners already listen to both, incoming control (via Coordinator) and data messages (via SUB). For a spectator, you just need to use that Listener.

I like the decoupling, which was the goal of LECO, as my own measurement software got hickups with concurrent read/writes, leading me to write LECO.

Indeed, this is already very well versed and I started using it to get the data.

@BenediktBurger: To your experience, what's the overhead of going through leco compared to communicating directly as it is now?

This proposal also reminds me of YAQ which I don't like because you always have your instruments hanging around and no direct control. It goes in the direction of other framework like tango, bluesky and we have to be carefull of what/who we target with pymodaq. Can we do a visio conf on the subject to exchange our views? @Ashwolaa do you want to chair it?

I was not aware of these softwares, the tango looks neat ^^
The mental picture I had looks indeed fairly close to a SCADA type (https://en.wikipedia.org/wiki/SCADA).
I would be happy to chair a visio and exchange more about it. Here is a evento to decide on a date.

https://evento.renater.fr/survey/leco-implementation-vmhcsw4v