Repository navigation
Conversation
|
Thanks for the update, @ai94iq, and for caring about test quality! Moving the example to a project copy of the resource and the shorter description are both good improvements. The core idea, proving that a test can actually fail, is something we'd really like agents to do. I think it lands best in a slightly different shape, and I'd love for you to take it there in this PR. What I noticed
A shape that would work wellCould you rework this PR into a short addition to ### Prove a new test can fail
Before you call a new or changed test done, break the behaviour it checks, not the expected value:
change the input, the test data or the code under test so the result must differ. Run only that
test with `robotcode robot -bl "<longname>"` and confirm it fails for the reason you expect. Undo
the change and confirm it passes again.And a small example that shows the difference: *** Test Cases ***
Cart Total Includes Every Item
${total}= Calculate Total 2 4
Should Be Equal As Integers ${total} 6
# Proof that the test works: temporarily make Calculate Total drop the last item
# (or pass 2 and 5), run the test, see it fail, then undo.
# Not a proof: a second assertion that expects 7.Then:
AI assistanceWe published AI_POLICY.md after you opened this PR, so you couldn't have known about it. It now applies to all open PRs, including this one. When you update, could you add the "AI assistance" section from the current PR template to the description (one line is enough, or "none" if no tool was involved)? Please also put your own handle in If you see it differently, just say so here, and we'll talk it through. Looking forward to the next version! 🙂 PS: I merged #13, #14, #15 and #16 today. I made a few small adjustments myself so they could go in right away; each PR's last comment lists them. Thanks for those! 🎉 |
|
Hi @ai94iq, a quick heads-up: we released 5.0.0 today, and it changes two things for this PR.
The rest of my last comment still holds: the rule and the example in |
Replaces the rf-test-quality skill with a short rule in rf-language: break the behaviour a test checks, run it, see it fail, undo.
f06155d to
ab1b959
Compare
|
Thanks @manykarim, reworked as suggested: the rule and example now live in rf-language, and the separate skill, script, resource and test-count changes are gone. I ran |
The example called a keyword it never defined, so its dry run failed.
What this changes
Adds a short "Prove a new test can fail" section to the
rf-languageskill (agent workflow): break the behaviour a test checks, not its expected value, run only that test withrobotcode robot -bl, see it fail, undo. Includes the cart-total example.The separate
rf-test-qualityskill, its script, resource, eval files and the installer test-count changes are dropped.AI assistance
Claude Code (Claude Opus) made the edits and ran the checks. I reviewed the diff and the check output, and I have the right to submit it.
Before you open it
robotcode; gaps are reported upstream.python scripts/build-catalog.pyandpython scripts/build-agent-variants.py(no generated file changed).For the reviewer
Text-only change inside rf-language; maintainers per catalog.yaml: @manykarim. No eval task added.
Local checks: build-catalog --check, check-governance, validate-skills --channel all, check-plugin-references and docs voice pass.
pytest tests plugins --ignore=tests/eval: one failure,test_hook_scripts.py::test_scripts_exit_zero_on_pathological_inputs, which fails the same way onmainon Windows.