Skip to content

docs(rf-language): prove a new test can fail - #11

Open
ai94iq wants to merge 2 commits into
MarketSquare:mainfrom
ai94iq:feat/test-quality-mutants
Open

ai94iq wants to merge 2 commits into
MarketSquare:mainfrom
ai94iq:feat/test-quality-mutants

Conversation

@ai94iq

@ai94iq ai94iq commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

Adds a short "Prove a new test can fail" section to the rf-language skill (agent workflow): break the behaviour a test checks, not its expected value, run only that test with robotcode robot -bl, see it fail, undo. Includes the cart-total example.

The separate rf-test-quality skill, its script, resource, eval files and the installer test-count changes are dropped.

AI assistance

Claude Code (Claude Opus) made the edits and ran the checks. I reviewed the diff and the check output, and I have the right to submit it.

Before you open it

  • Robot Framework capabilities go through robotcode; gaps are reported upstream.
  • Generated files were regenerated with their command, not edited by hand: python scripts/build-catalog.py and python scripts/build-agent-variants.py (no generated file changed).

For the reviewer

Text-only change inside rf-language; maintainers per catalog.yaml: @manykarim. No eval task added.

Local checks: build-catalog --check, check-governance, validate-skills --channel all, check-plugin-references and docs voice pass. pytest tests plugins --ignore=tests/eval: one failure, test_hook_scripts.py::test_scripts_exit_zero_on_pathological_inputs, which fails the same way on main on Windows.

@ai94iq
ai94iq requested a review from manykarim as a code owner November 15, 2026 07:47
@ai94iq ai94iq changed the title feat: add rf-test-quality mutant checks skill feat(rf-core): add rf-test-quality mutant checks Nov 15, 2026
@ai94iq

ai94iq commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Merge chain: #11, then #12, then #13, then #14, then #15, then #16.

All six merge into main cleanly in this order (dry run on 2026-11-15).

Apart from #13 and #15, the PRs do not depend on each other.

@manykarim

Copy link
Copy Markdown
Collaborator

Thanks for the update, @ai94iq, and for caring about test quality! Moving the example to a project copy of the resource and the shorter description are both good improvements. The core idea, proving that a test can actually fail, is something we'd really like agents to do. I think it lands best in a slightly different shape, and I'd love for you to take it there in this PR.

What I noticed

  • What the mutant proves. In the example (SKILL.md, lines 30-35) the mutant is a second Should Be Equal As Integers with a wrong expected value. That shows the assertion keyword fails on a wrong value, but not that the test goes red when the code under test breaks.
  • The resource copy. assets/mutants.resource imports Library ../scripts/mutation_ledger.py (line 3). Robot Framework resolves that relative to the copied file, so in <project>/resources/ it looks for <project>/scripts/mutation_ledger.py, which the skill never copies.
  • Listing budget. Claude Code lists every skill description in each session within about 8,000 characters, so a new skill that triggers on every test edit takes room from all users. A rule inside an existing skill gets the same effect at no cost.

A shape that would work well

Could you rework this PR into a short addition to plugins/rf-core/skills/rf-language/SKILL.md? For example:

### Prove a new test can fail

Before you call a new or changed test done, break the behaviour it checks, not the expected value:
change the input, the test data or the code under test so the result must differ. Run only that
test with `robotcode robot -bl "<longname>"` and confirm it fails for the reason you expect. Undo
the change and confirm it passes again.

And a small example that shows the difference:

*** Test Cases ***
Cart Total Includes Every Item
    ${total}=    Calculate Total    2    4
    Should Be Equal As Integers    ${total}    6
# Proof that the test works: temporarily make Calculate Total drop the last item
# (or pass 2 and 5), run the test, see it fail, then undo.
# Not a proof: a second assertion that expects 7.

Then:

  • drop the new skill, the script, the resource and the test-count changes;
  • regenerate bundles/ with python scripts/build-catalog.py and name that command in the PR;
  • if you'd like an eval task for it, word the prompt like a normal request ("add a test for the cart total") and let the grader check whether the agent proved the test can fail.

AI assistance

We published AI_POLICY.md after you opened this PR, so you couldn't have known about it. It now applies to all open PRs, including this one. When you update, could you add the "AI assistance" section from the current PR template to the description (one line is enough, or "none" if no tool was involved)? Please also put your own handle in metadata.author wherever you add one; this PR says manykarim.

If you see it differently, just say so here, and we'll talk it through. Looking forward to the next version! 🙂

PS: I merged #13, #14, #15 and #16 today. I made a few small adjustments myself so they could go in right away; each PR's last comment lists them. Thanks for those! 🎉

@manykarim

Copy link
Copy Markdown
Collaborator

Hi @ai94iq, a quick heads-up: we released 5.0.0 today, and it changes two things for this PR.

  • bundles/ is gone. 5.0.0 removed the rf-agentskills bundle, so please drop the bundles/rf-agentskills/ files from the branch. That replaces the "regenerate bundles/" step from my last comment: python scripts/build-catalog.py is still the command to run, it just no longer writes a bundle.
  • One version for everything. Every plugin and the installer now share one version, and a PR never bumps it. If your change is user-visible, add a line under ## Unreleased in the root CHANGELOG.md, in the rf-core group.

The rest of my last comment still holds: the rule and the example in rf-language would be the best shape. Rebasing on the current main should clear the conflicts. Thanks! 🙂

Replaces the rf-test-quality skill with a short rule in rf-language: break
the behaviour a test checks, run it, see it fail, undo.
@ai94iq
ai94iq force-pushed the feat/test-quality-mutants branch from f06155d to ab1b959 Compare November 20, 2026 05:25
@ai94iq

ai94iq commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @manykarim, reworked as suggested: the rule and example now live in rf-language, and the separate skill, script, resource and test-count changes are gone. I ran python scripts/build-catalog.py and added the AI assistance section.

@ai94iq ai94iq changed the title feat(rf-core): add rf-test-quality mutant checks docs(rf-language): prove a new test can fail Nov 20, 2026
The example called a keyword it never defined, so its dry run failed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants