Perf/cache epoch version in ClarityDatabase by jacinta-stacks · Pull Request #6959 · stacks-network/stacks-core

jacinta-stacks · 2026-03-05T18:26:11Z

Cache get_clarity_epoch_version() in a field on ClarityDatabase to eliminate redundant KV store lookups (2-3 times per map-set, map-get?, map-insert, map-delete, var-set, and NFT operation (for admits() type checks).

The epoch key is only written through set_clarity_epoch_version(), which updates the cache
roll_back() clears the cache defensively, so a rolled-back set_clarity_epoch_version() cannot leave a stale value
at-block also invalidates the store so it is reset there
ClarityDatabase instances are short-lived (created/destroyed per closure in with_clarity_db, as_transaction, etc.)

Benchmark Results (`cargo bench --bench epoch_cache -p clarity`)

Benchmark	No cache	Cached	Change
map_set_get/50	455 µs	304 µs	-33.2%
map_set_get/200	1.79 ms	1.18 ms	-34.1%
var_set_get/50	229 µs	176 µs	-23.1%
var_set_get/200	867 µs	668 µs	-22.9%
call_heavy/50	78 µs	77 µs	~0% (no epoch lookups in this path)
call_heavy/200	273 µs	267 µs	~0% (no epoch lookups in this path)
map_insert_delete/50	585 µs	446 µs	-23.8%
map_insert_delete/200	2.44 ms	1.84 ms	-24.6%

Signed-off-by: Jacinta Ferrant <236437600+jacinta-stacks@users.noreply.github.com>

…epoch Signed-off-by: Jacinta Ferrant <236437600+jacinta-stacks@users.noreply.github.com>

codecov · 2026-03-05T19:31:47Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.06%. Comparing base (995ebd0) to head (a92bda2).
⚠️ Report is 25 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff              @@
##           develop    #6959       +/-   ##
============================================
+ Coverage    68.63%   80.06%   +11.42%     
============================================
  Files          412      412               
  Lines       219013   219023       +10     
  Branches       338      338               
============================================
+ Hits        150330   175363    +25033     
+ Misses       68683    43660    -25023

Files with missing lines	Coverage Δ
clarity/src/vm/database/clarity_db.rs	`80.98% <100.00%> (+1.96%)`	⬆️

... and 257 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 995ebd0...a92bda2. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

cylewitruk-stacks

Makes sense to me. Just added a few benchmark suggestions.

clarity/benches/epoch_cache.rs

cylewitruk-stacks · 2026-03-09T12:49:01Z

clarity/benches/epoch_cache.rs

+            b.iter_batched(
+                setup_store,
+                |marf| run(&parsed, marf),
+                BatchSize::SmallInput,


I think that SmallInput may not be the best hint together with setup_store() since it tells Criterion that the values are small and it's okay to create thousands of them up-front (i.e. thousands of SQLite in-memory handles). Maybe use PerIteration or LargeInput?

But otoh if you don't see large memory usage when running the benchmarks then it might be small enough to not matter, I'm not sure off the top of my head how "resource hungry" the MemoryBackingStore + its SQLite in-mem instance is.

clarity/benches/epoch_cache.rs

… into perf/cache-epoch-per-block

Signed-off-by: Jacinta Ferrant <236437600+jacinta-stacks@users.noreply.github.com>

cylewitruk-stacks · 2026-03-11T17:33:32Z

Do you think we could add a couple of defensive regression tests specifically around the rollback and at-block cases asserting the cached value is cleared? The rollback case is clear in the diff, but not at-block (where I'm most insecure) -- and it's anyway cheap insurance that future changes don't mess this up?

jcnelson

I think there could be a better way to do this: accept the epoch as an argument to ClarityDatabase::new() and ClarityDatabase::new_with_rollback_wrapper(). In all cases where we actually open the Clarity database on chainstate (i.e. not MemoryBackingStore), the epoch will be known to the caller. That could save you an additional MARF'ed read.

… into perf/cache-epoch-per-block

jacinta-stacks · 2026-03-11T23:35:40Z

I think there could be a better way to do this: accept the epoch as an argument to ClarityDatabase::new() and ClarityDatabase::new_with_rollback_wrapper(). In all cases where we actually open the Clarity database on chainstate (i.e. not MemoryBackingStore), the epoch will be known to the caller. That could save you an additional MARF'ed read.

This is quite a substantial change. Not sure its worth the effort but I will give it a try and you can assess the result :)

Signed-off-by: Jacinta Ferrant <236437600+jacinta-stacks@users.noreply.github.com>

… read per transaction Signed-off-by: Jacinta Ferrant <236437600+jacinta-stacks@users.noreply.github.com>

…parse_epoch

… remove unused burn_state_db arg and in call chain

federico-stacks · 2026-04-08T11:20:31Z

clarity/src/vm/database/clarity_db.rs

    /// Since Clarity did not exist in stacks 1.0, the lowest valid epoch ID is stacks 2.0.
    /// The instantiation of subsequent epochs may bump up the epoch version in the clarity DB if
    /// Clarity is updated in that epoch.


nit: this doc string belongs to get_clarity_epoch_version and not to parse_epoch

crc: 33bd8e3

federico-stacks · 2026-04-08T11:55:16Z

stackslib/src/clarity_vm/clarity.rs

This could benefit from using ClarityDatabase::read_epoch_from_store, as is done in other call sites such as read_only_connection_checked and eval_read_only

crc: a2593d2

This also made burn_state_db arg unused and could be removed also from the call chain

federico-stacks · 2026-04-08T12:07:52Z

clarity/src/vm/database/clarity_db.rs

+    /// `get_clarity_epoch_version()` call and updated by
+    /// `set_clarity_epoch_version()`.  The epoch never changes
+    /// mid-block except through `set_clarity_epoch_version()`
+    cached_epoch: Option<StacksEpochId>,


nit: improve docstring for methods involving cached_epoch (general comment)

crc: 2cb8150

federico-stacks · 2026-04-08T12:21:46Z

stackslib/src/clarity_vm/clarity.rs

+        ClarityDatabase::new(self, headers_db, burn_state_db, None)
+    }
+
+    fn as_clarity_db_with_epoch<'b>(


nit: missing docstring

crc: 9de52bb

federico-stacks · 2026-04-08T14:06:54Z

clarity/src/vm/database/clarity_db.rs

        store: &'a mut dyn ClarityBackingStore,
        headers_db: &'a dyn HeadersDB,
        burn_state_db: &'a dyn BurnStateDB,
+        cached_epoch: Option<StacksEpochId>,


improve test coverage for cache_epoch being set during construction

crc: 38e54d7

coveralls · 2026-04-08T15:03:53Z

Coverage Report for CI Build 24140301216

Coverage decreased (-38.9%) to 46.817%

Details

Coverage decreased (-38.9%) from the base build.
Patch coverage: 83 uncovered changes across 5 files (66 of 149 lines covered, 44.3%).
85999 coverage regressions across 334 files.

Uncovered Changes

File	Changed	Covered	%
clarity/src/vm/database/sqlite.rs	73	1	1.37%
stackslib/src/clarity_vm/clarity.rs	40	34	85.0%
clarity/src/vm/database/clarity_db.rs	29	27	93.1%
stackslib/src/clarity_vm/database/mod.rs	2	0	0.0%
clarity/src/vm/docs/mod.rs	1	0	0.0%

Coverage Regressions

85999 previously-covered lines in 334 files lost coverage.

Top 10 Files by Coverage Loss	Lines Losing Coverage	Coverage
stackslib/src/chainstate/stacks/db/transactions.rs	8051	7.28%
stackslib/src/chainstate/stacks/db/blocks.rs	4459	35.59%
stackslib/src/chainstate/stacks/transaction.rs	4311	9.87%
stackslib/src/chainstate/burn/db/sortdb.rs	3575	42.39%
stackslib/src/chainstate/stacks/boot/mod.rs	3429	12.78%
stackslib/src/net/chat.rs	3334	27.49%
stackslib/src/chainstate/burn/operations/leader_block_commit.rs	2465	16.34%
stackslib/src/net/db.rs	1942	29.17%
stacks-signer/src/signerdb.rs	1580	35.13%
stackslib/src/chainstate/stacks/index/test/node.rs	1317	0.0%

Coverage Stats


Relevant Lines:	217989
Covered Lines:	102056
Line Coverage:	46.82%
Coverage Strength:	12315981.16 hits per line

💛 - Coveralls

federico-stacks

Actually moving the PR back to draft due this finding

federico-stacks · 2026-04-13T11:51:01Z

stackslib/src/clarity_vm/clarity.rs

+            let mut clarity_db = datastore.as_clarity_db_with_epoch(
+                &NULL_HEADER_DB,
+                &NULL_BURN_STATE_DB,
+                epoch.epoch_id,
+            );


the change from commit 6fd5f32, breaks some unit tests from chainstate::stacks::db::blocks::test (like process_transaction_fee_check)

I need to investigate this further.

Cache epoch version in ClarityDatabase and benchmark it

3451d95

Signed-off-by: Jacinta Ferrant <236437600+jacinta-stacks@users.noreply.github.com>

jacinta-stacks requested review from benjamin-stacks, brice-stacks, cylewitruk-stacks and francesco-stacks March 5, 2026 18:26

Make sure that when we set the block via at-block, we invalidate the …

1f5d255

…epoch Signed-off-by: Jacinta Ferrant <236437600+jacinta-stacks@users.noreply.github.com>

cylewitruk-stacks reviewed Mar 9, 2026

View reviewed changes

jacinta-stacks added 2 commits March 10, 2026 13:21

Merge branch 'develop' of https://github.com/stacks-network/stacks-core…

532e325

… into perf/cache-epoch-per-block

CRC: use iter_batched_ref and PerIteration in benching

a92bda2

Signed-off-by: Jacinta Ferrant <236437600+jacinta-stacks@users.noreply.github.com>

jcnelson previously approved these changes Mar 11, 2026

View reviewed changes

jcnelson self-requested a review March 11, 2026 18:47

jcnelson reviewed Mar 11, 2026

View reviewed changes

Merge branch 'develop' of https://github.com/stacks-network/stacks-core…

5b1663b

… into perf/cache-epoch-per-block

jacinta-stacks dismissed jcnelson’s stale review via 7d0bdcf March 12, 2026 00:30

jacinta-stacks requested review from cylewitruk-stacks and jcnelson March 12, 2026 00:42

jacinta-stacks force-pushed the perf/cache-epoch-per-block branch from 7d0bdcf to 7eecb90 Compare March 12, 2026 03:08

jacinta-stacks marked this pull request as draft March 12, 2026 03:10

jacinta-stacks force-pushed the perf/cache-epoch-per-block branch from 7eecb90 to e1d2a21 Compare March 12, 2026 03:39

Add tests for epoch rollback and block_at functionality

6bfa9fc

Signed-off-by: Jacinta Ferrant <236437600+jacinta-stacks@users.noreply.github.com>

jacinta-stacks force-pushed the perf/cache-epoch-per-block branch from e1d2a21 to 6bfa9fc Compare March 12, 2026 19:38

CRC: pass known epoch to ClarityDatabase constructors to avoid a MARF…

6fd5f32

… read per transaction Signed-off-by: Jacinta Ferrant <236437600+jacinta-stacks@users.noreply.github.com>

federico-stacks self-assigned this Apr 8, 2026

federico-stacks added 4 commits April 8, 2026 14:41

chore: merge develop with conflicts

8301fab

crc: restore and improve docstring for get_clarity_epoch_version and …

33bd8e3

…parse_epoch

crc: makes get_clarity_db_epoch_version use read_epoch_from_store and…

a2593d2

… remove unused burn_state_db arg and in call chain

chore: improve docstrings involved with cached_epoch

2cb8150

chore: add as_clarity_db_with_epoch docstring

9de52bb

federico-stacks reviewed Apr 8, 2026

View reviewed changes

federico-stacks added 2 commits April 8, 2026 16:12

test: add coverage for cached_epoch and read_epoch_from_store

38e54d7

chore: add changelog entry

efbee17

federico-stacks marked this pull request as ready for review April 8, 2026 14:18

federico-stacks reviewed Apr 13, 2026

View reviewed changes

federico-stacks marked this pull request as draft April 13, 2026 11:52

Conversation

jacinta-stacks commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark Results (cargo bench --bench epoch_cache -p clarity)

Uh oh!

codecov bot commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

cylewitruk-stacks left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cylewitruk-stacks commented Mar 11, 2026

Uh oh!

jcnelson left a comment

Choose a reason for hiding this comment

Uh oh!

jacinta-stacks commented Mar 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coveralls commented Apr 8, 2026

Coverage Report for CI Build 24140301216

Coverage decreased (-38.9%) to 46.817%

Details

Uncovered Changes

Coverage Regressions

Coverage Stats

💛 - Coveralls

Uh oh!

federico-stacks left a comment

Choose a reason for hiding this comment

Uh oh!

federico-stacks Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jacinta-stacks commented Mar 5, 2026 •

edited

Loading

Benchmark Results (`cargo bench --bench epoch_cache -p clarity`)

codecov bot commented Mar 5, 2026 •

edited

Loading

federico-stacks Apr 13, 2026 •

edited

Loading