Implement multiple tiers caching with fallback and backfilling

[Felixoid]

This PR addresses multiple requests to have tiered caching.

Closes #30
Closes #1020
Closes #2493
Closes #2566

What is added:

• multilevel configuration for the caching chain
• config parameter for cache writing tolerance: ignore errors, fail on write to L0, fail on a write to any level
• added stats for each caching level
• documentation
• tests
• integration tests

[codecov-commenter]

Codecov Report

❌ Patch coverage is 89.50715% with 198 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.17%. Comparing base (7b751fc) to head (43089ab).

Files with missing lines	Patch %	Lines
src/cache/multilevel.rs	87.16%	67 Missing ⚠️
src/cache/multilevel_test.rs	93.99%	63 Missing ⚠️
src/config.rs	79.91%	46 Missing ⚠️
src/cache/cache.rs	70.73%	12 Missing ⚠️
src/cache/disk.rs	81.25%	6 Missing ⚠️
src/cache/readonly.rs	0.00%	3 Missing ⚠️
src/server.rs	90.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2581      +/-   ##
==========================================
+ Coverage   73.37%   74.17%   +0.80%     
==========================================
  Files          68       70       +2     
  Lines       37338    39198    +1860     
==========================================
+ Hits        27396    29077    +1681     
- Misses       9942    10121     +179     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Felixoid]

I've done quite a few features today, like control for write error handling, and added server stats for multilevel caching.

[sylvestre]

this is a huge patch, can you split a bit more? thanks

[Felixoid]

Probably I could take something out of cache.rs, but the rest is actually multilevel.rs, and 2/3 of 2k LoC are tests.

[Felixoid]

Rebased to the current main.

This PR now contains the multi-level cache feature exclusively. Most of the changes are the tests and documentation, which are around 70%.

[Felixoid]

Hey-hey @trxcllnt, yeah, that's awesome. Thank you for the insight!

The improvement for the Storage trait looks very good from a performance perspective. While preparing this PR, I didn't want to change the architect so much and didn't consider the zero-copy approach.

The main question now is whether it is worth a) implementing in this PR, b) postponing until the next one, or rather c) implementing as a foundation for the changes, checking it works, and then moving to a dedicated PR.

update: I revisit it, and, at least for now, the copying looks negligible. The cold-cache fillback usually should be more or less quick compared to other IO (disk write/network). We can improve it in follow-up PRs.

[codspeed-hq]

Merging this PR will not alter performance

✅ 60 untouched benchmarks
⏩ 4 skipped benchmarks¹

---

_{Comparing Felixoid:multiple-tiers-cache (43089ab) with main (7b751fc)}

Footnotes

1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[Felixoid]

Dear @AJIOB and @mathstuf, I understand this PR looks huge, but most of the changes are docs and tests.

I'd very much appreciate this PR moving forward. It should significantly improve our local ClickHouse builds.

[mathstuf]

Docs look good, but I would like to request one thing (feel free to punt for now though). I don't like that precedence is restricted to service types. For example, I may have:

• local disk
• NFS share ("disk")
• node-local Redis
• data center Redis
• local S3
• global S3

and want to interleave them in priority order. Basically: allow naming storages and then use the names in the chain configuration. Pure envvar configuration is probably lost, but maybe that's fine?

Overall, I love the idea, just asking for "one more thing" before things get baked in here. Again, feel free to ignore; it can probably be tacked on later too (by reserving all impl names or requiring a prefix on custom names).

[AJIOB]

In general, I like that changes.

There's awesome work, thanks, @Felixoid!

JFYI, you have a failed CUDA test. Please, recheck this point too.

[Felixoid]

Thanks a lot for the review!

@AJIOB: it looks like the CUDA tests were temporarily broken for an unknown reason. It was Segmentation fault in sudo apt-get update. Looks working fine now.

@mathstuf: as much as I like the idea, it looks like a next iteration for now. Probably, I'd like to work on it too. But it's for sure will be a design change for the configuration, and should be discussed in advance on how to proceed.

Now, the approach is simple and logical for the current configuration.

[Felixoid]

@sylvestre do you think we can proceed?

[drahnr]

Bar the shell scripts, the core implementation does look sound. Also took a look @trxcllnt which is a more minimal approach with 2 cache layers (or using nesting).
Since this PR already got reviews I suggest to move it along.

@sylvestre if you find the time, take a look, otherwise I'll do sometime next week and we should land it imho.

[Felixoid]

Thanks for the review!

I am on vacation RN, will try to find time to address the points somewhere around the following week.

[Felixoid]

@drahnr I addressed all points. Sorry that it took so long. I tried not to touch the laptop for work