Aller au contenu
login
arrow_backRetour aux issues
prometheus/prometheus #14926

feat: Avoid decay of our micro-benchmarks; test them on CI

ecoDébutant verifiedCNCF help wanted priority/P2 component/tests area/ci-cd

descriptionDescription

### Proposal As per https://github.com/prometheus/prometheus/pull/14925, our code tends to change, but microbenchmarks (`func BenchmarkXYZ(b *testing.B)`) decay. They always build, but the they decay "semantically" by essentially have a broken flows (timeouts, deadlocks), they code they benchmark is suddenly wrongly configured (e.g. so you don't benchmark success case anymore but failures/incorrect execution). Microbenchmarks decay because (currently) we don't require to re-run the benchmarks on every change of the related code. 1) For this reason in the past I wrote for my "Efficient Go" book a quick [`testorbench` structure](https://github.com/efficientgo/core/blob/main/testutil/testorbench.go#L26) that allows running benchmarks "as a test with a N == 1" just to test runnability. `testorbench` also allows doing more tests on `!tb.IsBenchmark()`, as you don't want to do detailed asserts during benchmark time as it's waste of time (success criteria should be deterministic) and you want to measure execution not tests. You can see example use [here](https://github.com/bwplotka/benchmarks/blob/main/benchmarks/metrics-streaming/benchmark_test.go#L38). 2) One alternative would be to have some `go test -run ^$ -bench .* -benchtime 1x` just to at least see if things are runnable on CI. 3) Finally, if we get the nightly microbenchmark compared to e.g. day before build setup, that would keep the whole pipeline fresh perhaps and give us meaningful data. Maybe that's a solution too (: See [how Vitess is doing this](https://benchmark.vitess.io/). ## Acceptance Criteria * Our micro-benchmark code is not decaying e.g. it's semantics and flow is updated when the related code is updated. cc @bboreham @cstyan Honestly (3) would be nice to do!
codeOuvre sur GitHub