Repository navigation
Conversation
Merging this PR will degrade performance by 63.35%2 warnings
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
48afad2 to
8a03135
Compare
50e19c6 to
c8191de
Compare
[Medium impact] The change reaches Go benchmark collection and the profile files it produces. The PR appears safe to merge; adding a compressed-file test is a non-blocking improvement.
|
Walltime benchmarks run many rounds, but the results only carried summary stats. Report the distribution as well: - Each benchmark's stats gain `percentiles_ns`, p1 to p99 of the time per iteration. They use statrs' quantile formula on the rounds sorted once: statrs runs an O(n) selection per call, so 99 calls took 5.2 s on 10M rounds against 0.38 s for one sort. q1, median and q3 now come from them: q1 and q3 are unchanged, and the median, mean and stdev can differ in the last floating-point bit. - go-runner writes the time and iteration count of every round, in execution order, to `walltime_rounds/<pid>.json.gz` after `results/<pid>.json`. b.Loop() saves each iteration as a round, so a benchmark can run millions of them: above 100,000, it keeps a uniformly random subsample, still in execution order, and then has fewer rounds than `stats.rounds`. On a 126,490-round b.Loop() benchmark, the 100,000 kept rounds matched the full p1 to p99 within 0.74%, and gzip at its fastest level took the file from 802 KB to 231 KB. That level runs 11 times faster than the default on 2.5M rounds, for a file 20 to 30% larger. Results files keep parsing with older consumers: the new stats field is optional, and the rounds live in their own file. `getrandom` was already built through `tempfile`, and `flate2` was already a dependency. Refs COD-4241
c8191de to
ca3b96a
Compare
Report the distribution of walltime measurements, not only their summary stats. Same change as CodSpeedHQ/codspeed-rust#205, in go-runner's copy of
walltime_results.rs.results/<pid>.jsongainpercentiles_ns, p1 to p99 of the time per iteration. They use statrs' quantile formula on the rounds sorted once, instead of an O(n) statrs selection per percentile.q1_ns,median_nsandq3_nsnow come from the same list: q1 and q3 are unchanged, and the median, mean and stdev can differ in the last floating-point bit.results/<pid>.json, go-runner writes the time and iteration count of every round, in execution order, towalltime_rounds/<pid>.json.gz:{"benchmarks": [{"uri": "example/fib_test.go::BenchmarkFibonacci20_Loop", "times_per_round_ns": [35606, 35571, ...], "iters_per_round": [1, 1, ...]}]}b.Loop()saves each iteration as a round, so a benchmark can run millions of them. Above 100,000 rounds, a benchmark keeps a uniformly random subsample, still in execution order. A benchmark was subsampled when it has fewertimes_per_round_nsthanstats.rounds.Tested with
BenchmarkFibonacci20_Loopfromexample/at-benchtime 5sundercodspeed run: 126,490 rounds, 100,000 kept, and the kept rounds match the full p1 to p99 within 0.74%. The rounds file of that run is 802 KB of JSON, 231 KB gzipped.Writing the rounds file costs time in
bench_collect_results, mostly serializing and compressing up to 100,000 rounds per benchmark. gzip runs at its fastest level: on the 2.5M rounds ofbench_collect_results[100000, 25], it took 0.28 s locally against 3.0 s at the default level, for a file 20 to 30% larger (231 KB instead of 178 KB on the run above).Compatibility:
percentiles_nsis optional, and matches the fieldrunner-sharedaccepts in CodSpeedHQ/codspeed#630. The rounds live in their own file, so current consumers of the results file are unaffected, and the snapshots don't change since they redactstats.getrandomwas already built throughtempfile, andflate2was already a dependency.The percentile and sampling code is a copy of codspeed-rust's; a
TODO(COD-4241)asks to call thecodspeedcrate instead.Refs COD-4241