tsdb/agent: Checkpoint based on Series in Memory

[kgeckhart]

Proposal

The agent currently uses the same Checkpoint implementation as all other parts of prometheus,

prometheus/tsdb/wlog/checkpoint.go

Lines 87 to 90 in 61aa828

	// Checkpoint creates a compacted checkpoint of segments in range [from, to] in the given WAL.
	// It includes the most recent checkpoint if it exists.
	// All series not satisfying keep, samples/tombstones/exemplars below mint and
	// metadata that are not the latest are dropped.

The checkpoint serves three purposes for agent mode,

1. Populates the agent db stripeSeries with known series + last sample timestamps on startup
2. Populate series caches in queue_manager on startup
3. Pruning the series caches in queue_manager after a new checkpoint is created
4. Not applicable for agent mode yet and might be dropped Most recently metadata for a series

This is an incredibly small subset of the data vs what is persisted in a checkpoint which includes, series which exist in the WAL, samples above mint, float and regular histogram samples above mint, exemplars above mint, and latest metadata. In order to create a checkpoint with all these records we re-read the current checkpoint + all segments. This is a lot of overhead given all the data we require for the checkpoint is currently in memory between stripeSeries and the deleted series in agent db.

I propose we introduce another checkpoint implementation which could look something like,

type ActiveSeries interface {
    Ref() chunks.HeadSeriesRef
    Labels() labels.Labels
    LastSampleTimestamp() int64
}

// Checkpoint creates an unindexed checkpoint containing record.RefSeries and 
// record.RefSample for ActiveSeries and  a record.RefSeries for the recentlyDeleted series. 
func Checkpoint(logger *slog.Logger, w *WL, seriesIter iter.Seq[ActiveSeries], recentlyDeleted []chunks.HeadSeriesRef)

that could be driven by the data we currently have in memory which would,

1. Reduce the overhead of taking a checkpoint
2. Reduce the overhead of queue_manager reading a checkpoint as checkpoints will be smaller
3. Improve startup times/resource usage due to smaller checkpoint sizes

I did a quick implementation of this in Grafana Alloy where it shrunk a 214MB checkpoint by 56% down to 137MB, with the following improvements to creating a checkpoint + loading a checkpoint

              │ old-create.txt │           new-create.txt           │
              │     sec/op     │   sec/op     vs base               │
Checkpoint-11     3477.6m ± 7%   913.3m ± 6%  -73.74% (p=0.002 n=6)

              │ old-create.txt │            new-create.txt            │
              │      B/op      │     B/op       vs base               │
Checkpoint-11   2717.25Mi ± 0%   11.52Mi ± 11%  -99.58% (p=0.002 n=6)

              │ old-create.txt  │           new-create.txt           │
              │    allocs/op    │ allocs/op   vs base                │
Checkpoint-11   34087723.5 ± 0%   325.0 ± 1%  -100.00% (p=0.002 n=6)

                │ baseline-load.txt │           new-load.txt            │
                │      sec/op       │   sec/op    vs base               │
LoadLargeWAL-11          4.195 ± 2%   1.105 ± 5%  -73.67% (p=0.002 n=6)

                │ baseline-load.txt │            new-load.txt             │
                │       B/op        │     B/op      vs base               │
LoadLargeWAL-11        2.001Gi ± 1%   1.204Gi ± 0%  -39.83% (p=0.002 n=6)

                │ baseline-load.txt │            new-load.txt            │
                │     allocs/op     │  allocs/op   vs base               │
LoadLargeWAL-11         35.22M ± 0%   30.76M ± 0%  -12.66% (p=0.002 n=6)

[bwplotka]

👍🏽

It's interesting that there's a pattern where we fallback from using WAL back to memory. I think that makes sense in general, use only one memory space for things if we can, given it can be loaded from the persisted WAL if needed (vs treating queue manager and head totally isolated)

[kgeckhart]

It's interesting that there's a pattern where we fallback from using WAL back to memory. I think that makes sense in general, use only one memory space for things if we can, given it can be loaded from the persisted WAL if needed (vs treating queue manager and head totally isolated)

Yeah I think the incredibly loose coupling they have today adds a lot of complexity and reducing that where we can make some trade-offs could go a long way! I was thinking a bit about what I proposed and I think it's actually breaking the current concept of checkpointing. The current checkpoint is truly a compacted representation from the segments / existing checkpoint so it makes sense that is the "starting" segment.

The new checkpointing functionality is not this though it's a point in time snapshot of what is in memory which is decoupled from the segments. I adjusted the proposed function to drop the index and would likely do an "un-indexed" checkpoint in this case. If this still makes sense I'll see how the implementation looks / behaves for the agent which is more straightforward/lower stakes. Then we can look at if it makes sense to apply to head as well and what it might mean for checkpoint'ing in general.

[bboreham]

I think you are assuming the current behaviour where replay is started from after the restart, rather than replaying all un-sent data that was collected in the WAL. This is listed as an open issue #8809.

Assuming that issue was fixed, you would need a more complex solution to this issue. Maybe checkpointing all data to disk which has not been acknowledged over remote-write?

[kgeckhart]

Assuming that issue was fixed, you would need a more complex solution to this issue. Maybe checkpointing all data to disk which has not been acknowledged over remote-write?

I think it will depend on what ends up being the accepted solution to that issue. The proposal I put forth (prometheus/proposals#72) is based on #17616 which would allow us to safely assume the segments contain everything we need to send and the checkpoint exists for persisting in-memory WAL state for a restart.

Even if we do not move forward with my proposal, #17616, is still a problem for unsent data. If my read on the flows is correct the data would never be sent because the checkpoint is only read on startup (which today is only used to drive loading the in-memory WAL state) and async by the watcher so QueueManger to manage its series cache.

We could do a replay which has to consider the checkpoint but I think the complexity and resource overhead would be higher. I can try to put together some figures/analysis to show but generally it feels awkward to take a hit re-reading from disk as much as we do today vs using what we have in memory and keeping read IO for restart and sending.

[beorn7]

Hello from the bug scrub!

This might be covered by prometheus/proposals#72. Leaving it open for now.