Skip to content

feat(rf-core): add the rf-bdd skill - #46

Open
manykarim wants to merge 1 commit into
mainfrom
change/add-rf-bdd-skill-v5
Open

manykarim wants to merge 1 commit into
mainfrom
change/add-rf-bdd-skill-v5

Conversation

@manykarim

Copy link
Copy Markdown
Collaborator

Summary

Adds rf-bdd to rf-core: writing and reviewing behaviour-driven specifications in native .robot and .resource files (OpenSpec change add-rf-bdd-skill).

  • Skill: SKILL.md (169 lines) with 11 house rules, Gherkin input, 11 gotchas; 7 references; assets/robocop-bdd.toml; two runnable example projects (resource strategy, World library) on a fake ATM service.
  • Verified: the examples analyse, dry-run and pass on RF 7.4.2 and 7.5.1 (and Python 3.8 with RF 7.3.2), and lint clean on Robocop 8.2.10 and 9.4. All 11 gotchas were reproduced on both version sets (trap-verification.md).
  • rf-language: points to rf-bdd; the casing gotcha follows the project's Robocop configuration; corrects "a comma-separated --select is rejected" to "selects nothing".
  • Routing: the context hook and rf-test-architect (plus Codex/OpenCode variants) route BDD work to rf-bdd.
  • Evals: sut-bdd-library fixture (base, steps, review), a BDD grader with 30 tests and golden solutions, five tasks, a trigger set (11 positive, 11 negative, 4 holdout). One-line cli.py preflight fix: tasks are selected by fixture root (sut-bdd-library/base → sut-bdd-library).
  • CHANGELOG ## Unreleased, README and rf-core README updated; skill count 10.

Not in this PR

Tasks 1.1, 5.5 and 5.6 (live trigger baseline and tuning, CI-model task baselines) follow in a separate PR. A local spot check with Haiku 5.5 loaded rf-bdd in 3 of 3 runs; the late-return suites passed (5/5, 10/10) with no .feature files.

Test plan

  • Full pytest locally: all pass except the 4 Browser tests that need Playwright
  • build-catalog --check, governance, content rules, validate-skills, plugin references, drift, docs voice, security scan
  • openspec validate --all --strict
  • CI

AI assistance

Written with Claude Code (Claude Opus 5.5); the maintainer reviewed the change and directed the scope. Reference porting and the eval fixture were done by subagents and checked by the tests above.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP

Adds rf-bdd for behaviour-driven specifications in native .robot files:
house rules, Gherkin mapping, step vocabularies, scenario state, a
Robocop configuration, and two runnable example projects (resource and
World-library strategies), verified on RF 7.4.2 and 7.5.1 and on
Robocop 8.2.10 and 9.4.

- rf-language points to rf-bdd; its casing gotcha follows the project's
  Robocop configuration; a comma-separated --select selects nothing
  (it is not rejected).
- The context hook and rf-test-architect route BDD work to rf-bdd.
- Evals: the sut-bdd-library fixture (base, steps, review variants),
  a BDD grader with tests and golden solutions, five tasks and a
  trigger set. The preflight selects tasks by fixture root.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FHEmzL3ErAhGQykKWDx2wP
@github-actions

Copy link
Copy Markdown

Skill Evaluation: rf-mobile (narrow, treatment, N=3, z-ai/glm-5.3-flash)

Model: z-ai/glm-5.3-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.18
Harness estimate (OpenRouter list prices) 0.02
Credit remaining on the key 39.684829278

Gate

gate: PASS (1 task(s) checked)

rf-skill-eval report

Runs: 3 · pass: 3 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.02

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-appium-login-01 rf-appium narrow treatment 3 100% 100% 0 218645 4676 15.3 44.0 0.0073 (est.) no
narrow-appium-login-01 rf-appium narrow baseline (stored) 3 100% 100% 0 126666 2566 10.3 30.9 0.0043 (est.) –
narrow-appium-login-01 rf-appium narrow Δ T−B +0 pp +91979 +2110 +5.0 +13.2 +0.0030

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-appium 1 100% 100% +0 pp +91979 +5.0 +0.0030

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-appium-login-01 loaded_rf_appium_skill: 3/3

Adversarial tier (reported only, never gates)

None.

@github-actions

Copy link
Copy Markdown

Skill Evaluation: rf-desktop (narrow, treatment, N=3, z-ai/glm-5.3-flash)

Model: z-ai/glm-5.3-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.22
Harness estimate (OpenRouter list prices) 0.03
Credit remaining on the key 39.651180168

Gate

gate: PASS (1 task(s) checked)

rf-skill-eval report

Runs: 3 · pass: 2 · fail: 1 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.03 · turn-limit stops: 1

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-platynui-calculator-01 rf-platynui narrow treatment 3 67% 67% 0 310407 4892 16.3 (1 at limit) 52.4 0.0100 (est.) yes
narrow-platynui-calculator-01 rf-platynui narrow baseline (stored) 3 100% 100% 0 361178 4420 19.7 (1 at limit) 109.6 0.0116 (est.) –
narrow-platynui-calculator-01 rf-platynui narrow Δ T−B -33 pp -50771 +472 -3.3 -57.2 -0.0016

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-platynui 1 67% 100% -33 pp -50771 -3.3 -0.0016

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-platynui-calculator-01 loaded_rf_platynui_skill: 3/3

Adversarial tier (reported only, never gates)

None.

@github-actions

Copy link
Copy Markdown

Skill Evaluation: rf-web (narrow, treatment, N=3, z-ai/glm-5.3-flash)

Model: z-ai/glm-5.3-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.29
Harness estimate (OpenRouter list prices) 0.05
Credit remaining on the key 39.574078708

Gate

gate: PASS (2 task(s) checked)

rf-skill-eval report

Runs: 6 · pass: 6 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.05

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-browser-login-01 rf-browser narrow treatment 3 100% 100% 0 358293 2763 20.0 50.6 0.0113 (est.) no
narrow-browser-login-01 rf-browser narrow baseline (stored) 6 100% 100% 0 70371 1454 6.7 22.1 0.0024 (est.) –
narrow-browser-login-01 rf-browser narrow Δ T−B +0 pp +287923 +1310 +13.3 +28.5 +0.0089
narrow-selenium-login-01 rf-selenium narrow treatment 3 100% 100% 0 191717 2197 12.7 31.2 0.0063 (est.) no
narrow-selenium-login-01 rf-selenium narrow baseline (stored) 6 100% 100% 0 72006 801 8.0 12.7 0.0024 (est.) –
narrow-selenium-login-01 rf-selenium narrow Δ T−B +0 pp +119711 +1395 +4.7 +18.5 +0.0039

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-browser 1 100% 100% +0 pp +287923 +13.3 +0.0089
rf-selenium 1 100% 100% +0 pp +119711 +4.7 +0.0039

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-browser-login-01 loaded_rf_browser_skill: 3/3
  • narrow-selenium-login-01 loaded_rf_selenium_skill: 3/3

Adversarial tier (reported only, never gates)

None.

@github-actions

Copy link
Copy Markdown

Skill Evaluation: rf-api (narrow, treatment, N=3, z-ai/glm-5.3-flash)

Model: z-ai/glm-5.3-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 0.30
Harness estimate (OpenRouter list prices) 0.06
Credit remaining on the key 39.563985783

Gate

gate: PASS (2 task(s) checked)

rf-skill-eval report

Runs: 6 · pass: 6 · fail: 0 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.06 · turn-limit stops: 1

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-requests-users-01 rf-requests narrow treatment 3 100% 100% 0 257122 3747 17.7 45.1 0.0085 (est.) no
narrow-requests-users-01 rf-requests narrow baseline (stored) 6 100% 100% 0 78154 836 8.0 12.3 0.0025 (est.) –
narrow-requests-users-01 rf-requests narrow Δ T−B +0 pp +178968 +2911 +9.7 +32.9 +0.0060
narrow-restinstance-schema-01 rf-restinstance narrow treatment 3 100% 100% 0 317913 2943 17.7 (1 at limit) 47.5 0.0100 (est.) no
narrow-restinstance-schema-01 rf-restinstance narrow baseline (stored) 6 100% 100% 0 142111 1992 11.8 25.9 0.0047 (est.) –
narrow-restinstance-schema-01 rf-restinstance narrow Δ T−B +0 pp +175802 +952 +5.8 +21.5 +0.0053

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-requests 1 100% 100% +0 pp +178968 +9.7 +0.0060
rf-restinstance 1 100% 100% +0 pp +175802 +5.8 +0.0053

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-requests-users-01 loaded_rf_requests_skill: 3/3
  • narrow-restinstance-schema-01 loaded_rf_restinstance_skill: 3/3

Adversarial tier (reported only, never gates)

None.

@github-actions

Copy link
Copy Markdown

Skill Evaluation: rf-core (narrow, treatment, N=3, z-ai/glm-5.3-flash)

Model: z-ai/glm-5.3-flash (provider openrouter, turn-limit factor 1.5)

OpenRouter spend

USD
Key spend during this job (includes concurrent jobs) 1.07
Harness estimate (OpenRouter list prices) 0.56
Credit remaining on the key 38.794013894

Gate

gate: REBASELINE-NEEDED (16 task(s) checked)
- [rebaseline-needed] narrow-bdd-extend-vocabulary-01: no stored baseline entry
- [rebaseline-needed] narrow-bdd-greenfield-01: no stored baseline entry
- [rebaseline-needed] narrow-bdd-outline-01: no stored baseline entry
- [rebaseline-needed] narrow-bdd-review-01: no stored baseline entry

Trigger gate

gate: REBASELINE-NEEDED (1 task(s) checked)
- [rebaseline-needed] triggers:rf-bdd: no stored trigger baseline

rf-skill-eval trigger report

Model z-ai/glm-5.3-flash · skill-listing budget default · a query is triggered when its trigger rate (loads / runs) is >= its set's threshold. TP/FP/TN/FN per split; train and validation are reported separately (tune on train only).

Skill Train TP/FP/TN/FN Train P Train R Train Acc Val TP/FP/TN/FN Val P Val R Val Acc Stored Val Acc
rf-bdd – – – – 4/0/4/0 100% 100% 100% –

Description visibility

Sessions whose skill listing showed the skill's description (the rest listed it by name only). Own counts the sessions of the skill's own queries, All every session in this result.

Skill Own sessions All sessions
rf-appium – 24/24
rf-bdd 24/24 24/24
rf-browser – 24/24
rf-language – 24/24
rf-platynui – 24/24
rf-python-library – 24/24
rf-requests – 24/24
rf-restinstance – 24/24
rf-selenium – 24/24
rf-setup – 24/24

Failing queries

None.

rf-skill-eval report

Runs: 48 · pass: 45 · fail: 3 · incomplete: 0 · skipped checks: 0 · estimated cost (OpenRouter list prices): $0.54 · turn-limit stops: 2

Pass rate = runs whose gate result is pass / runs attempted. Deltas are treatment − baseline over outcome checks only.

Tasks

Task Skill Tier Arm Runs Pass rate Outcome pass Incomplete Input tok Output tok Turns Duration s Cost $ Flaky
narrow-bdd-extend-vocabulary-01 rf-bdd narrow treatment 3 100% 100% 0 715990 13299 27.0 117.8 0.0232 (est.) no
narrow-bdd-extend-vocabulary-01 rf-bdd narrow Δ T−B unavailable unavailable unavailable unavailable unavailable unavailable
narrow-bdd-greenfield-01 rf-bdd narrow treatment 3 100% 100% 0 583457 15994 23.7 150.5 0.0195 (est.) no
narrow-bdd-greenfield-01 rf-bdd narrow Δ T−B unavailable unavailable unavailable unavailable unavailable unavailable
narrow-bdd-outline-01 rf-bdd narrow treatment 3 100% 100% 0 1518209 36787 39.7 309.5 0.0491 (est.) no
narrow-bdd-outline-01 rf-bdd narrow Δ T−B unavailable unavailable unavailable unavailable unavailable unavailable
narrow-bdd-review-01 rf-bdd narrow treatment 3 0% 0% 0 319852 5672 20.3 50.6 0.0106 (est.) no
narrow-bdd-review-01 rf-bdd narrow Δ T−B unavailable unavailable unavailable unavailable unavailable unavailable
narrow-language-bdd-01 rf-language narrow treatment 3 100% 100% 0 192598 1125 13.3 16.6 0.0061 (est.) no
narrow-language-bdd-01 rf-language narrow baseline (stored) 6 100% 100% 0 79812 848 7.5 13.0 0.0026 (est.) –
narrow-language-bdd-01 rf-language narrow Δ T−B +0 pp +112787 +277 +5.8 +3.6 +0.0035
narrow-language-data-driven-01 rf-language narrow treatment 3 100% 100% 0 201214 2978 11.7 30.0 0.0068 (est.) no
narrow-language-data-driven-01 rf-language narrow baseline (stored) 6 83% 83% 0 114378 1513 10.0 24.1 0.0038 (est.) –
narrow-language-data-driven-01 rf-language narrow Δ T−B +17 pp +86836 +1464 +1.7 +5.9 +0.0030
narrow-language-embedded-01 rf-language narrow treatment 3 100% 100% 0 179371 3663 11.3 37.0 0.0062 (est.) no
narrow-language-embedded-01 rf-language narrow baseline (stored) 6 100% 100% 0 106643 2098 9.2 25.9 0.0037 (est.) –
narrow-language-embedded-01 rf-language narrow Δ T−B +0 pp +72728 +1565 +2.2 +11.2 +0.0025
narrow-language-env-varfiles-01 rf-language narrow treatment 3 100% 100% 0 479210 6265 22.3 (2 at limit) 69.1 0.0152 (est.) no
narrow-language-env-varfiles-01 rf-language narrow baseline (stored) 6 100% 100% 0 193964 5167 17.2 (1 at limit) 52.4 0.0066 (est.) –
narrow-language-env-varfiles-01 rf-language narrow Δ T−B +0 pp +285247 +1098 +5.2 +16.7 +0.0086
narrow-language-suite-setup-tags-01 rf-language narrow treatment 3 100% 100% 0 246102 2565 17.3 38.2 0.0079 (est.) no
narrow-language-suite-setup-tags-01 rf-language narrow baseline (stored) 6 100% 100% 0 112928 1551 13.2 21.0 0.0037 (est.) –
narrow-language-suite-setup-tags-01 rf-language narrow Δ T−B +0 pp +133173 +1014 +4.2 +17.2 +0.0042
narrow-non-rf-control-01 plugin narrow treatment 3 100% 100% 0 37058 197 2.3 3.3 0.0013 (est.) no
narrow-non-rf-control-01 plugin narrow baseline (stored) 6 100% 100% 0 26311 95 2.0 3.2 0.0009 (est.) –
narrow-non-rf-control-01 plugin narrow Δ T−B +0 pp +10747 +101 +0.3 +0.1 +0.0004
narrow-python-library-flaky-listener-01 rf-python-library narrow treatment 3 100% 100% 0 190874 2475 13.3 26.7 0.0062 (est.) no
narrow-python-library-flaky-listener-01 rf-python-library narrow baseline (stored) 6 100% 100% 0 139834 3160 11.5 34.2 0.0046 (est.) –
narrow-python-library-flaky-listener-01 rf-python-library narrow Δ T−B +0 pp +51039 -685 +1.8 -7.4 +0.0016
narrow-python-library-inventory-scope-01 rf-python-library narrow treatment 3 100% 100% 0 155330 2156 10.0 28.0 0.0052 (est.) no
narrow-python-library-inventory-scope-01 rf-python-library narrow baseline (stored) 6 100% 100% 0 246142 3910 16.5 (1 at limit) 49.6 0.0082 (est.) –
narrow-python-library-inventory-scope-01 rf-python-library narrow Δ T−B +0 pp -90812 -1754 -6.5 -21.6 -0.0030
narrow-python-library-mode-literal-01 rf-python-library narrow treatment 3 100% 100% 0 238793 2449 14.0 29.1 0.0076 (est.) no
narrow-python-library-mode-literal-01 rf-python-library narrow baseline (stored) 6 100% 100% 0 128381 2979 10.3 34.9 0.0044 (est.) –
narrow-python-library-mode-literal-01 rf-python-library narrow Δ T−B +0 pp +110411 -531 +3.7 -5.8 +0.0032
narrow-rf-injection-positive-01 plugin narrow treatment 3 100% 100% 0 87640 947 6.3 13.4 0.0028 (est.) no
narrow-rf-injection-positive-01 plugin narrow baseline (stored) 6 100% 100% 0 64147 1338 4.8 15.4 0.0022 (est.) –
narrow-rf-injection-positive-01 plugin narrow Δ T−B +0 pp +23493 -391 +1.5 -2.0 +0.0006
narrow-rf-mcp-execute-01 plugin narrow treatment 3 100% 100% 0 230479 1276 12.0 24.5 0.0075 (est.) no
narrow-rf-mcp-execute-01 plugin narrow baseline (stored) 6 100% 100% 0 145726 933 8.5 17.4 0.0049 (est.) –
narrow-rf-mcp-execute-01 plugin narrow Δ T−B +0 pp +84753 +343 +3.5 +7.0 +0.0026
narrow-setup-requests-01 rf-setup narrow treatment 3 100% 100% 0 160916 1763 11.7 25.2 0.0053 (est.) no
narrow-setup-requests-01 rf-setup narrow baseline (stored) 6 100% 100% 0 96690 839 8.5 18.0 0.0032 (est.) –
narrow-setup-requests-01 rf-setup narrow Δ T−B +0 pp +64226 +924 +3.2 +7.2 +0.0021

Skills

Skill Tasks Treatment outcome pass Baseline outcome pass Δ outcome pass Δ input tok Δ turns Δ cost $
rf-bdd 4 75% – unavailable unavailable unavailable unavailable
rf-language 5 100% 97% +3 pp +138154 +3.8 +0.0044
rf-python-library 3 100% 100% +0 pp +23546 -0.3 +0.0006
rf-setup 1 100% 100% +0 pp +64226 +3.2 +0.0021

Skipped / errored checks

None.

Incomplete runs

None.

Process checks (treatment arm, not in deltas)

  • narrow-bdd-extend-vocabulary-01 loaded_rf_bdd_skill: 3/3
  • narrow-bdd-greenfield-01 loaded_rf_bdd_skill: 3/3
  • narrow-bdd-outline-01 loaded_rf_bdd_skill: 3/3
  • narrow-bdd-review-01 loaded_rf_bdd_skill: 3/3
  • narrow-language-bdd-01 loaded_rf_language_skill: 3/3
  • narrow-language-data-driven-01 loaded_rf_language_skill: 3/3
  • narrow-language-embedded-01 loaded_rf_language_skill: 3/3
  • narrow-language-env-varfiles-01 loaded_rf_language_skill: 3/3
  • narrow-language-suite-setup-tags-01 loaded_rf_language_skill: 3/3
  • narrow-non-rf-control-01 bounded_tool_calls: 3/3
  • narrow-non-rf-control-01 no_libdoc_invocation: 3/3
  • narrow-non-rf-control-01 no_rf_mcp_calls: 3/3
  • narrow-python-library-flaky-listener-01 loaded_rf_python_library_skill: 3/3
  • narrow-python-library-inventory-scope-01 loaded_rf_python_library_skill: 3/3
  • narrow-python-library-mode-literal-01 loaded_rf_python_library_skill: 3/3
  • narrow-rf-injection-positive-01 leveraged_rf_lookup: 3/3
  • narrow-rf-mcp-execute-01 rf_mcp_before_write: 3/3
  • narrow-rf-mcp-execute-01 rf_mcp_called_at_least_twice: 3/3
  • narrow-rf-mcp-execute-01 rf_mcp_had_successful_result: 3/3
  • narrow-setup-requests-01 used_uv_add: 3/3

Adversarial tier (reported only, never gates)

None.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant