Skip to content

Rework performance benchmarks around serializer-only slices#619

Merged
brenbar merged 4 commits into
mainfrom
copilot/rework-performance-tests
May 27, 2026
Merged

Rework performance benchmarks around serializer-only slices#619
brenbar merged 4 commits into
mainfrom
copilot/rework-performance-tests

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 27, 2026

The performance suite was dominated by NATS transport and too many iterations, which made runs slow and obscured serializer behavior. This reworks the harness to measure only Serializer.serialize/Serializer.deserialize size and timing, while making targeted slices cheap to run.

  • Scope the benchmark to codec work

    • Removed local/remote NATS setup and all transport latency measurements from the harness.
    • Replaced request/response round-trip benchmarking with direct request/response serialization and deserialization measurement in Python, TypeScript, and Java.
    • Kept Telepact JSON, Telepact binary, Telepact packed binary, protobuf, and plain JSON comparisons.
  • Reduce run cost and support slices

    • Lowered steady-state sampling to 20 iterations.
    • Limited warmup to a single binary cache warmup for Telepact binary modes.
    • Added filters for --languages, --data-shapes, --collection-shapes, and --methods so slices can be run without executing the full matrix.
  • Align shared outputs with the new model

    • Simplified shared metrics to codec-only timings:
      • requestSerializationTimeNs
      • requestDeserializationTimeNs
      • responseSerializationTimeNs
      • responseDeserializationTimeNs
    • Removed network metadata from generated summaries and CSV/JSON outputs.
    • Regenerated the committed benchmark summary artifacts from the updated harness.
  • Trim perf-only dependencies

    • Removed NATS dependencies from the performance test projects.
    • Added a small Java serializer factory helper so the benchmark can construct client/server serializers directly without transport plumbing.
  • Refresh docs and examples

    • Updated the binary performance guide to describe the serializer-only benchmark and the new results.
    • Documented slice-oriented usage for quick reruns.
cd test/performance

uv run python -m perf_harness.run \
  --languages python \
  --data-shapes all-numbers \
  --collection-shapes huge-list \
  --methods telepact-binary telepact-packed-binary

Copilot AI and others added 4 commits May 27, 2026 01:13
Co-authored-by: brenbar <12563144+brenbar@users.noreply.github.com>
Co-authored-by: brenbar <12563144+brenbar@users.noreply.github.com>
Co-authored-by: brenbar <12563144+brenbar@users.noreply.github.com>
Co-authored-by: brenbar <12563144+brenbar@users.noreply.github.com>
@brenbar brenbar marked this pull request as ready for review May 27, 2026 01:37
@brenbar brenbar added this pull request to the merge queue May 27, 2026
Merged via the queue into main with commit 193b367 May 27, 2026
17 checks passed
@brenbar brenbar deleted the copilot/rework-performance-tests branch May 27, 2026 01:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants