Skip to content

Say when a run reached only ungated groups instead of "every gate met" - #716

Merged
TonsOfFun merged 1 commit into
activeagents:mainfrom
yorzi:fix/ungated-gates-message
Nov 17, 2026
Merged

TonsOfFun merged 1 commit into
activeagents:mainfrom
yorzi:fix/ungated-gates-message

Conversation

@yorzi

@yorzi yorzi commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

What this does

Report::Gate#met? is always true for an ungated group. So a report whose every result sits in ungated groups has only met rows. In that case GateCheck#message, which assert_meets_pass_gates and the RSpec matcher print, reads "every pass gate met", and evals run --wait prints Gates: every gate met (N)., even when every answer failed:

gates:   ["experimental · m: 0/1 (0%) passed, not gated"]
passed?: true
GateCheck#message: "every pass gate met"

No gate checked anything, but the output says every gate passed.

This keeps every verdict and changes only the wording:

  • GateCheck#message: "every group the run covered is ungated; no answer was held to a bar".
  • CLI: Gates: every group reached is ungated (N); nothing was held to a bar.
  • docs/framework/evaluations.md and docs/framework/cli.md describe the case, and the CLI page points at --min-pass-rate for a run that should still fail CI.

Tests:

  • Fail before the change: test/evals/gate_check_test.rb (an all-ungated, all-failing report is met and says nothing was held to a bar) and test/cli/cli_test.rb (such a run exits 0 with the new line and without "every gate met").
  • Unchanged and passing: the empty-report message, the gated messages, EvaluationPassGatesTest's "an ungated set is reported, not gated" (gates_met true), the TestHelper and RSpec matcher tests.

Decision for the maintainer

This PR is also a proposal. Please confirm it, redirect it, or close it.

I first tried the stricter reading: an all-ungated report does not pass, like the existing rule for a report with no gate. I reverted it because it breaks two contracts this repo states. The docs say "a group with gated: false is reported and never fails the run", and EvaluationPassGatesTest asserts gates_met == true for a run of one ungated set. Under the stricter rule, an exploratory catalog set run with evals run --wait would start exiting 4.

If you would rather have the stricter semantics, they are a three-line follow-up: Report#passed? and EvaluationRun#gates_met require at least one gated row, and print_gates drops its new branch. I'd be glad to switch this PR to that.

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Performance improvement

Scope check

  • I read the contributing guide
  • This fits activeagent's stated focus
  • This isn't application-specific logic that belongs in user code
  • This benefits most users, not just my use case

Related issue

None filed. Follows the per-group gates of #676, #682 and #704.

Quality check

  • The project's hooks and linters pass (bin/rubocop on the changed files)
  • The new tests fail before this change and pass after it
  • The full test suite passes for what this touches: framework suite (Rails 8.1) 2585 runs, 0 failures, 0 errors; actionagent/test/evaluation_pass_gates_test.rb green. CliContractTest errors identically on an untouched main in this environment (rack-session undefined method 'id' for Hash)
  • I updated documentation where needed
  • I didn't edit generated files by hand

Ownership

  • I understand every line of this change and can answer questions about it

API changes

  • Breaking change
  • New public methods or classes
  • Changed method signatures
  • No API changes (one message string changes for an all-ungated report)

An ungated group's gate row is met without being held to a bar. A report
whose every result sat in ungated groups therefore had only met rows, and
GateCheck#message (what assert_meets_pass_gates and the RSpec matcher
print) said "every pass gate met", as did `evals run --wait` ("Gates:
every gate met"), even when every answer had failed.

The verdict is kept: the docs promise an ungated group never fails the
run, and a catalog set marked `gated: false` is tested to report gates_met
true. Only the wording changes. GateCheck says every group the run covered
is ungated and no answer was held to a bar, and the CLI prints "Gates:
every group reached is ungated (N); nothing was held to a bar." The
evaluations and CLI guides note it, and point at --min-pass-rate for a
run that should still fail CI.
@TonsOfFun
TonsOfFun merged commit b1ddf6c into activeagents:main Nov 17, 2026
18 checks passed
TonsOfFun added a commit that referenced this pull request Nov 17, 2026
@TonsOfFun TonsOfFun added this to the 1.9.9 milestone Nov 17, 2026
@TonsOfFun TonsOfFun mentioned this pull request Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants