Add adapter for `criterion-cycles-per-byte`

[Pr0methean]

I have a Rust benchmark that uses criterion_cycles_per_byte, and I've tried running in the Bencher CLI with all the following commands, but they all give "Are you sure (rust-criterion|json|magic) is the right adapter?" and appear on my dashboard as runs with no metrics.

bencher run --token [TOKEN] --project pr0methean-s-rand-triplemix cargo +nightly bench -p benches --features reproducibility_cross_platform,bench_include_threadrng
bencher run --token [TOKEN] --project pr0methean-s-rand-triplemix cargo +nightly bench -p benches --features reproducibility_cross_platform,bench_include_threadrng -- --output-format bencher
bencher run --adapter json --token [TOKEN] --project pr0methean-s-rand-triplemix cargo +nightly bench -p benches --features reproducibility_cross_platform,bench_include_threadrng --message-format=json
bencher run --adapter rust_criterion --token [TOKEN] --project pr0methean-s-rand-triplemix cargo +nightly bench -p benches --features reproducibility_cross_platform,bench_include_threadrng
bencher run --adapter json --token [TOKEN] --project pr0methean-s-rand-triplemix cargo +nightly bench -p benches --features reproducibility_cross_platform,bench_include_threadrng -- --output-format bencher

How do I fix this? The benchmark code is at https://github.com/Pr0methean/TripleMixPrng/blob/7ec278d7719c43202671d2753c52a4fa9fa46a4b/benches/src/benchmarks.rs. Please don't force me to revert to using wall-clock time on platforms that criterion_cycles_per_byte supports. The noise from CPU-cache contention is bad enough even without any noise from thermal throttling or preemptive multitasking, and I was hoping to use Relative Continuous Benchmarking, and/or the "control group" that bench_include_threadrng enables, to mitigate even the former noise to the point that the results would finally be usable!

[epompeii]

Hey @Pr0methean, thank you for using Bencher, and I'm sorry you're running into trouble.

I have a Rust benchmark that uses criterion_cycles_per_byte

Are you using this criterion_cycles_per_byte crate?

If so, this is not a benchmark harness that Bencher currently supports. However, I am open to a PR to add support for it!

I would want it to be a separate adapter from rust_criterion named rust_criterion_cycles_per_byte.

If waiting for this PR to land is not an option, you can also check out the rust_gungraun adapter.

[Pr0methean]

Actually, it would probably be much more broadly useful if your team developed a plug-in for Criterion that would output BMF, because then it would work with all Criterion plugins that define metrics, such as criterion_linux_perf and criterion_perf_events.

It would also be a lot less confusing if your Rust-specific documentation mentioned that you were a different Bencher than the one that Criterion's --output-format bencher setting is named after.

[epompeii]

Actually, it would probably be much more broadly useful if your team developed a plug-in for Criterion that would output BMF, because then it would work with all Criterion plugins that define metrics, such as criterion_linux_perf and criterion_perf_events.

If that is something you would be willing to create and maintain, then I would highly encourage you to contribute it to the Criterion ecosystem! Several other benchmark harnesses have started to output BMF.

However, this is not something that Bencher would be willing to create and maintain ourselves at this time. We would however, be open to adding adapters for any popular Criterion plugins, such as criterion-cycles-per-byte and criterion_perf_events (#763).

I've update this issue to track adding criterion-cylces-per-byte support.

It would also be a lot less confusing if your Rust-specific documentation mentioned that you were a different Bencher than the one that Criterion's --output-format bencher setting is named after.

Yes, the name collision is unfortunate.

From the Criterion docs:

To change the CLI output format, use cargo bench -- --output-format <name>. Supported output formats are:

• criterion - Use Criterion's normal output format
• bencher - An output format similar to the output produced by the bencher crate or nightly libtest benchmarks. Though this provides less information than the criterion format, it may be useful to support external tools that can parse this output.

Criterion Command-Line Options

I would argue that naming this output format bencher was a bit of a confusing choice for Criterion, as it is just a stable port of the libtest benchmark harness and its output. That is, libtest would have probably have been a clearer name. (@bheisler any insight on the design choice here?)

I have been hopeful that a standard JSON schema for benchmark results would come to fruition in the Rust ecosystem. However, it seems like BMF may be the best bet for now. Something like: --output-format bmf

[Pr0methean]

Update: I'm attempting to build a converter for the --message-format=json format, but I won't be able to test it until bheisler/criterion.rs#904 is fixed. The work in progress is at https://github.com/pr0methean/criterion-json-to-bmf. The work in progress is at https://github.com/pr0methean/criterion-json-to-bmf.