out of order get_packets response

[sunws4me]

Describe the bug

packets returned via get_packets from a subscription are not always in the correct order.

To Reproduce

create microservice
request all minor frame packets for format X
validate minor frame cnt is in order

Expected behavior

tlm returned from get_packets should always be in the same order as received from source

Screenshots

No response

OS

alma9x

OpenC3 COSMOS Version

7.1.0

Browser

chrome

[crawlchat]

I can see the issue clearly in the source code. Let me confirm the root cause.

The bug is in get_packets in tlm_api.rb (lines 485–492). Here's the problematic flow:

xread.each do |topic, data|   # iterates per-topic
  data.each do |id, msg_hash|
    ...
    packets << msg_hash.merge(json_hash)  # appends all packets for topic 1, then all for topic 2, etc.
  end
end

When get_packets is called with a subscription that covers multiple packet streams (even if they are the same "packet name" but spread across topics or shards), the results from xread are grouped per-topic — so all packets from topic A are appended first, then all packets from topic B. This breaks chronological ordering.

In your case with minor frames across format variants, if those end up on different Redis/Valkey stream topics, the interleaving is lost and they come back topic-by-topic rather than in receive order.

📋 More details on the root cause

Topic.read_topics returns a hash keyed by topic (xread). The outer loop iterates topics, and the inner loop iterates messages within that topic. There is no cross-topic sort by time or received_time before building the packets array.

Since each topic's messages are in order within themselves, but topics are not interleaved by timestamp, the final packets array is only ordered correctly when subscribing to a single topic.

Suggested workaround (until this is fixed upstream):

After calling get_packets, sort the returned packets by their time (or received_time) field:

id, packets = get_packets(id)
packets.sort_by! { |p| p['time'].to_i }

Or in Python:

id_, packets = get_packets(id_)
packets.sort(key=lambda p: int(p['time']))

Suggested fix in tlm_api.rb (lines ~492):

# After the xread loop, sort all packets by received time before returning
packets.sort_by! { |p| p['time'].to_i }
return lookup.to_a.join(SUBSCRIPTION_DELIMITER), packets

Could you help clarify a couple of things so the engineering team can fully assess this?

1. Are you subscribing to multiple packet names (e.g., MINOR_FRAME_FORMAT_1, MINOR_FRAME_FORMAT_2, etc.) in a single subscribe_packets call, or a single packet?
2. Are these targets on different db shards (multi-shard COSMOS deployment)?

This will help confirm whether the out-of-order issue is cross-topic or potentially something else.

[jmthomas]

The recommended workaround might be the permanent solution:

id = subscribe_packets([['INST', 'HEALTH_STATUS'], ['INST', 'ADCS']])
wait(2)
id, packets = get_packets(id)
packets.sort(key=lambda p: int(p['time']))  # -> sort!!!
for packet in packets:
    print(f"{packet['PACKET_TIMESECONDS']}: {packet['target_name']} {packet['packet_name']}")

Sorting in the backend for people that aren't requesting more than 1 packet could become a heavy handed operation depending on the number of packets they are requesting.

At a minimum we should document this behavior and the workaround.

[sunws4me]

more documentation on the logic is always good. I think one issue with the workaround is that the user probably really has to pool the responses for a hot second then process data, only processing up until something gets out of order, then wait for more continue processing or report the dropped packet if its important.

[jmthomas]

We could add an argument to get_packets to sort the batch that is returned. But the next batch could still have something that wouldn't be sorted between batches.

[sunws4me]

based on the current capability, I would just document the behavior and leave to the implementer.