Skip to content

feat(go-runner): report walltime percentiles and rounds - #70

Open
fargito wants to merge 1 commit into
mainfrom
feat/walltime-percentiles-and-rounds
Open

fargito wants to merge 1 commit into
mainfrom
feat/walltime-percentiles-and-rounds

Conversation

@fargito

@fargito fargito commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Report the distribution of walltime measurements, not only their summary stats. Same change as CodSpeedHQ/codspeed-rust#205, in go-runner's copy of walltime_results.rs.

  • Percentiles: each benchmark's stats in results/<pid>.json gain percentiles_ns, p1 to p99 of the time per iteration. They use statrs' quantile formula on the rounds sorted once, instead of an O(n) statrs selection per percentile. q1_ns, median_ns and q3_ns now come from the same list: q1 and q3 are unchanged, and the median, mean and stdev can differ in the last floating-point bit.
  • Rounds: after results/<pid>.json, go-runner writes the time and iteration count of every round, in execution order, to walltime_rounds/<pid>.json.gz:
    {"benchmarks": [{"uri": "example/fib_test.go::BenchmarkFibonacci20_Loop", "times_per_round_ns": [35606, 35571, ...], "iters_per_round": [1, 1, ...]}]}
    b.Loop() saves each iteration as a round, so a benchmark can run millions of them. Above 100,000 rounds, a benchmark keeps a uniformly random subsample, still in execution order. A benchmark was subsampled when it has fewer times_per_round_ns than stats.rounds.

Tested with BenchmarkFibonacci20_Loop from example/ at -benchtime 5s under codspeed run: 126,490 rounds, 100,000 kept, and the kept rounds match the full p1 to p99 within 0.74%. The rounds file of that run is 802 KB of JSON, 231 KB gzipped.

Writing the rounds file costs time in bench_collect_results, mostly serializing and compressing up to 100,000 rounds per benchmark. gzip runs at its fastest level: on the 2.5M rounds of bench_collect_results[100000, 25], it took 0.28 s locally against 3.0 s at the default level, for a file 20 to 30% larger (231 KB instead of 178 KB on the run above).

Compatibility: percentiles_ns is optional, and matches the field runner-shared accepts in CodSpeedHQ/codspeed#630. The rounds live in their own file, so current consumers of the results file are unaffected, and the snapshots don't change since they redact stats. getrandom was already built through tempfile, and flate2 was already a dependency.

The percentile and sampling code is a copy of codspeed-rust's; a TODO(COD-4241) asks to call the codspeed crate instead.

Refs COD-4241

@codspeed

codspeed Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 63.35%

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ Unable to measure the execution of 1 benchmark

This benchmark measured no execution time, so its result is not comparable and counts as unchanged. This usually happens when the compiler removes the code under test.

Prevent compiler optimizations

See affected benchmarks

❌ 15 regressed benchmarks
✅ 35 untouched benchmarks
⏩ 1 skipped benchmark1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ bench_collect_results[100000, 25] 42.7 ms 753.2 ms -94.33%
❌ bench_collect_results[100000, 10] 20.9 ms 305.3 ms -93.15%
❌ bench_collect_results[100000, 5] 19.2 ms 162.2 ms -88.19%
❌ bench_collect_results[500000, 25] 257.9 ms 1,038.6 ms -75.17%
❌ bench_collect_results[500000, 10] 121.9 ms 440.5 ms -72.32%
❌ bench_collect_results[500000, 5] 103.2 ms 278.1 ms -62.9%
❌ bench_collect_results[1000000, 25] 523.6 ms 1,296.1 ms -59.6%
❌ bench_collect_results[1000000, 10] 246.1 ms 559.4 ms -56%
❌ bench_collect_results[1000000, 5] 205.7 ms 376.1 ms -45.3%
❌ bench_collect_results[5000000, 25] 2.6 s 3.7 s -29.25%
❌ bench_collect_results[5000000, 10] 1.3 s 1.7 s -23.82%
❌ bench_collect_results[10000000, 25] 5.6 s 6.9 s -18.99%
❌ bench_collect_results[10000000, 10] 2.6 s 3.1 s -18.57%
❌ bench_collect_results[5000000, 5] 1 s 1.3 s -16.98%
❌ bench_collect_results[10000000, 5] 2.1 s 2.4 s -14.02%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing feat/walltime-percentiles-and-rounds (ca3b96a) with main (d004aae)

Open in CodSpeed

Footnotes

  1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports. ↩

@fargito
fargito force-pushed the feat/walltime-percentiles-and-rounds branch from 48afad2 to 8a03135 Compare November 18, 2026 14:47
@fargito fargito changed the title feat(go-runner): report walltime percentiles and raw rounds feat(go-runner): report walltime percentiles and rounds Nov 18, 2026
@fargito
fargito force-pushed the feat/walltime-percentiles-and-rounds branch 2 times, most recently from 50e19c6 to c8191de Compare November 19, 2026 09:40
@fargito
fargito marked this pull request as ready for review November 19, 2026 10:18
@greptile-apps

greptile-apps Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The change reaches Go benchmark collection and the profile files it produces.

The PR appears safe to merge; adding a compressed-file test is a non-blocking improvement.

Fix All in Claude CodeFindings

  1. P2 Compressed rounds lack test coverage ▶
Fix with agent prompt
### Issue 1
go-runner/src/lib.rs:66-74
**Compressed rounds lack test coverage.** The new tests check the rounds payload in memory, but none opens `walltime_rounds/<pid>.json.gz`. Existing integration tests read only `results/`, so a wrong output path or incomplete gzip file could pass unnoticed. Add a test that calls `collect_walltime_results`, decompresses the saved file, and checks its benchmark URI and paired round values.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The runner now reports walltime percentiles and saves individual rounds in a separate compressed file.

  • Benchmark results now include percentiles from p1 to p99.
  • Benchmark collection saves round measurements in a compressed file.

Reviews (1) · Last reviewed commit: "feat(go-runner): report walltime percent..." · Reviewed by Greptile

Comment thread go-runner/src/lib.rs Resolved
Walltime benchmarks run many rounds, but the results only carried summary
stats. Report the distribution as well:

- Each benchmark's stats gain `percentiles_ns`, p1 to p99 of the time per
  iteration. They use statrs' quantile formula on the rounds sorted once:
  statrs runs an O(n) selection per call, so 99 calls took 5.2 s on 10M
  rounds against 0.38 s for one sort. q1, median and q3 now come from them:
  q1 and q3 are unchanged, and the median, mean and stdev can differ in the
  last floating-point bit.
- go-runner writes the time and iteration count of every round, in
  execution order, to `walltime_rounds/<pid>.json.gz` after
  `results/<pid>.json`. b.Loop() saves each iteration as a round, so a
  benchmark can run millions of them: above 100,000, it keeps a uniformly
  random subsample, still in execution order, and then has fewer rounds
  than `stats.rounds`. On a 126,490-round b.Loop() benchmark, the 100,000
  kept rounds matched the full p1 to p99 within 0.74%, and gzip at its
  fastest level took the file from 802 KB to 231 KB. That level runs 11
  times faster than the default on 2.5M rounds, for a file 20 to 30%
  larger.

Results files keep parsing with older consumers: the new stats field is
optional, and the rounds live in their own file. `getrandom` was already
built through `tempfile`, and `flate2` was already a dependency.

Refs COD-4241
@fargito
fargito force-pushed the feat/walltime-percentiles-and-rounds branch from c8191de to ca3b96a Compare November 19, 2026 10:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant