Skip to content

Use the rsc implementation of the patricia trie#126

Merged
mhutchinson merged 2 commits into
transparency-dev:mainfrom
mhutchinson:rsc
May 20, 2026
Merged

Use the rsc implementation of the patricia trie#126
mhutchinson merged 2 commits into
transparency-dev:mainfrom
mhutchinson:rsc

Conversation

@mhutchinson
Copy link
Copy Markdown
Contributor

@mhutchinson mhutchinson commented May 11, 2026

This required changing the API structs. We have tagged a v0.1.0 that
operators of the old code should pin to.

This implementation has comparable disk and memory implementations:

go test -bench=BenchmarkBuild -benchmem ./vindex

goos: linux
goarch: amd64
pkg: github.com/transparency-dev/incubator/vindex
cpu: AMD EPYC 7B12
BenchmarkBuild_InMemory-24    	       1	1006305591 ns/op	13464448 B/op	  244751 allocs/op
BenchmarkBuild_OnDisk-24      	       1	1001417280 ns/op	14245768 B/op	  261960 allocs/op
PASS
ok  	github.com/transparency-dev/incubator/vindex	22.217s

Compared with old implementation:

New: BenchmarkBuild_InMemory-24    	       1	1006305591 ns/op	13464448 B/op	  244751 allocs/op
Old: BenchmarkBuild_InMemory-24                1        1004559670 ns/op        38192304 B/op     588597 allocs/op

New: BenchmarkBuild_OnDisk-24      	       1	1001417280 ns/op	14245768 B/op	  261960 allocs/op
Old: BenchmarkBuild_OnDisk-24                  1        4810209571 ns/op        2268211632 B/op  1541229 allocs/op

Given that the disk performance is now so good, and superior for
restarts, the vindex implementations have been changed to use
persistence by default (though support a flag override to use the
memory version).

Comparing real world performance of this vs the previous version is most
useful. For a full read of SumDB:

# Old version - memory only (persisted version failed)
OUTPUT_LOG_PRIVATE_KEY= go run ./vindex/cmd/sumdbindex --storage_dir      280.92s user 131.52s system 176% cpu 3:54.03 total

# New version, persisted
OUTPUT_LOG_PRIVATE_KEY= go run ./vindex/cmd/sumdbindex --storage_dir      164.35s user 105.55s system 147% cpu 3:03.23 total
# New version, memory
OUTPUT_LOG_PRIVATE_KEY= go run ./vindex/cmd/sumdbindex --storage_dir      320.98s user 134.93s system 180% cpu 4:13.08 total

Comment thread vindex/map.go Fixed
@mhutchinson mhutchinson force-pushed the rsc branch 5 times, most recently from 26bf979 to f62fdd7 Compare May 19, 2026 15:12
@mhutchinson mhutchinson requested a review from AlCutter May 19, 2026 15:12
Comment thread vindex/map.go Fixed
This required changing the API structs. We have tagged a v0.1.0 that
operators of the old code should pin to.

This implementation has comparable disk and memory implementations:

```
go test -bench=BenchmarkBuild -benchmem ./vindex

goos: linux
goarch: amd64
pkg: github.com/transparency-dev/incubator/vindex
cpu: AMD EPYC 7B12
BenchmarkBuild_InMemory-24    	       1	1006305591 ns/op	13464448 B/op	  244751 allocs/op
BenchmarkBuild_OnDisk-24      	       1	1001417280 ns/op	14245768 B/op	  261960 allocs/op
PASS
ok  	github.com/transparency-dev/incubator/vindex	22.217s
```

Compared with old implementation:

```
New: BenchmarkBuild_InMemory-24    	       1	1006305591 ns/op	13464448 B/op	  244751 allocs/op
Old: BenchmarkBuild_InMemory-24                1        1004559670 ns/op        38192304 B/op     588597 allocs/op

New: BenchmarkBuild_OnDisk-24      	       1	1001417280 ns/op	14245768 B/op	  261960 allocs/op
Old: BenchmarkBuild_OnDisk-24                  1        4810209571 ns/op        2268211632 B/op  1541229 allocs/op
```

Given that the disk performance is now so good, and superior for
restarts, the vindex implementations have been changed to use
persistence by default (though support a flag override to use the
memory version).

Comparing real world performance of this vs the previous version is most
useful. For a full read of SumDB:

```
OUTPUT_LOG_PRIVATE_KEY= go run ./vindex/cmd/sumdbindex --storage_dir      280.92s user 131.52s system 176% cpu 3:54.03 total

OUTPUT_LOG_PRIVATE_KEY= go run ./vindex/cmd/sumdbindex --storage_dir      164.35s user 105.55s system 147% cpu 3:03.23 total
OUTPUT_LOG_PRIVATE_KEY= go run ./vindex/cmd/sumdbindex --storage_dir      320.98s user 134.93s system 180% cpu 4:13.08 total
```
Comment thread vindex/client/client.go
@mhutchinson mhutchinson merged commit e874678 into transparency-dev:main May 20, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants