tsdb/agent: Prevent unread segments from being truncated

[kgeckhart]

Proposal

Today the tsdb/agent truncates the the WAL segments based on the assumption,

prometheus/tsdb/agent/db.go

Line 676 in f50ff0a

// The lower two-thirds of segments should contain mostly obsolete samples.

which is not verified before truncation occurs.

Due to this it's really hard to determine how much downtime can be tolerated in a remote write configuration since it's a factor of TruncateFrequency + rate of data in. This leads to a much higherTruncateFrequencythan is really necessary and much larger WALs that must be fully replayed on startup. Internally we run with a 15 minute interval as we are okay with trading off downtime tolerance for less memory usage.

I would propose remote.Storage expose the ability to subscribe to be notified when a segment changes. This would be called after all current queues have read past a segment, sample implementation.

After segment notifications are working the tsdb/agent could subscribe and truncate based on segments that have been fully read. At this point we could consider dropping the default TruncateFrequency allowing for smaller WALs.

I'm not sure how much of this, if any, is applicable for the tsdb proper

[aviralgarg05]

I would like to solve this issue

[beorn7]

Hello from the bug scrub!

@aviralgarg05 let's first get some idea if this is the right way to go and how it should look like in detail.

@jesusvazquez what do you think about this?

[mishraa-G]

Hey @beorn7 @jesusvazquez I’ve been working on optimizing the WAL truncation for Agent mode.
Basically, I switched it from the time-based heuristic to actual consumption tracking.
The Watcher now tracks the max timestamp for each segment, and QueueManager uses its highestSentTimestamp to figure out which segments are fully sent and safe to delete. The Agent just polls for that safe segment index and truncates up to there.
I added some tests to verify it handles the segments correctly and falls back to the old way if needed.
Does that approach sound good to you? If so, I can open up the PR.

[beorn7]

I was only the bug scrubber here. I am actually not qualified enough to comment on any WAL truncation in Agent mode. @jesusvazquez would be more qualified (or might name somebody who is).

[bwplotka]

Sorry for lag.

It's been sometime I touched agent algorithm, but we might need to rely on the agent more over time.

This idea makes sense to me -- I am surprised we didn't implement this earlier. I thought the main advantage of agent is the ability to kill data that was already sent and we can build robust communication pipeline between agent and RW queues for that.

Segment based tracking might be a nice mechanism, given it likely doesn't make sense to truncate segments partially.

I am not sure if I'm fan of the initial implementation though (grafana@5d1e37d). Notably, let's try to not leak too much detail to too many components. WAL watching is aware of segments, so great to track and communicate segment changes from there. But why queue_manager needs to know about segment changes? It wasn't aware before about segments, right?

[beorn7]

Hello from the bug scrub!

This is now being discussed in prometheus/proposals#72 .