Repository navigation
Say when a run reached only ungated groups instead of "every gate met" - #716
Merged
Merged
Conversation
An ungated group's gate row is met without being held to a bar. A report
whose every result sat in ungated groups therefore had only met rows, and
GateCheck#message (what assert_meets_pass_gates and the RSpec matcher
print) said "every pass gate met", as did `evals run --wait` ("Gates:
every gate met"), even when every answer had failed.
The verdict is kept: the docs promise an ungated group never fails the
run, and a catalog set marked `gated: false` is tested to report gates_met
true. Only the wording changes. GateCheck says every group the run covered
is ungated and no answer was held to a bar, and the CLI prints "Gates:
every group reached is ungated (N); nothing was held to a bar." The
evaluations and CLI guides note it, and point at --min-pass-rate for a
run that should still fail CI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Report::Gate#met?is always true for an ungated group. So a report whose every result sits in ungated groups has only met rows. In that caseGateCheck#message, whichassert_meets_pass_gatesand the RSpec matcher print, reads "every pass gate met", andevals run --waitprintsGates: every gate met (N)., even when every answer failed:No gate checked anything, but the output says every gate passed.
This keeps every verdict and changes only the wording:
GateCheck#message: "every group the run covered is ungated; no answer was held to a bar".Gates: every group reached is ungated (N); nothing was held to a bar.docs/framework/evaluations.mdanddocs/framework/cli.mddescribe the case, and the CLI page points at--min-pass-ratefor a run that should still fail CI.Tests:
test/evals/gate_check_test.rb(an all-ungated, all-failing report is met and says nothing was held to a bar) andtest/cli/cli_test.rb(such a run exits 0 with the new line and without "every gate met").EvaluationPassGatesTest's "an ungated set is reported, not gated" (gates_mettrue), the TestHelper and RSpec matcher tests.Decision for the maintainer
This PR is also a proposal. Please confirm it, redirect it, or close it.
I first tried the stricter reading: an all-ungated report does not pass, like the existing rule for a report with no gate. I reverted it because it breaks two contracts this repo states. The docs say "a group with
gated: falseis reported and never fails the run", andEvaluationPassGatesTestassertsgates_met == truefor a run of one ungated set. Under the stricter rule, an exploratory catalog set run withevals run --waitwould start exiting 4.If you would rather have the stricter semantics, they are a three-line follow-up:
Report#passed?andEvaluationRun#gates_metrequire at least one gated row, andprint_gatesdrops its new branch. I'd be glad to switch this PR to that.Type of change
Scope check
Related issue
None filed. Follows the per-group gates of #676, #682 and #704.
Quality check
bin/rubocopon the changed files)actionagent/test/evaluation_pass_gates_test.rbgreen.CliContractTesterrors identically on an untouchedmainin this environment (rack-sessionundefined method 'id' for Hash)Ownership
API changes