Repository navigation
fix(probes): give tool_calling's simulated service its other endpoints and refine its judges - #169
Merged
Merged
Conversation
The tool simulator and its judge read only the called tool's spec, so an ID one tool returned could take a form another offered tool does not accept: a listing tool returning `pet_8f3c1d92` where the detail tool takes an integer `pet_id`, and every follow-up call then failed its schema check. Both now read the service's other endpoints, the other offered tools with their parameter types, one per line (`_other_endpoints`). An ID or other value a response returns has the type and form those endpoints take for it, and the judge fails a response that returns one they don't take. The tool_calling prompt version is v1.12. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
The simulated service's clock is UTC. In the simulator's prompt and its judge's, opening hours, market sessions and other times tied to a place follow the local time there, converted from that UTC time, so whether a shop is open or a market is in session agrees with the place's local time. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
…w items An identical call returns the same data, which also had a tool that returns random or newly generated items, such as a random image or a generated password, repeat its earlier result. Such a tool now returns new items each time, in the simulator's rules and in the judge's consistency criterion. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
… the response The response judge fails a value the tool computes when it is wrong. A field that only describes the response itself, such as a count of the items it lists or an offset derived from its other fields, is not grounds for failure, so the judge's attempts go to the values a call is about. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
…d real tool gaps The per-turn assistant judge fails a reply that says it cannot do something one of the assistant's tools does. Declining for a legal, safety or privacy reason, or saying accurately that no offered tool does what is asked, is not that failure. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
…e acted on now The opening request's check also fails a request that can't be acted on yet, because it waits on something that hasn't happened, and one that needs a file or photo the user can't give as a link, so the simulated user writes it again. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issue
None.
What changed and why
tool_calling's simulated service now sees the service's other endpoints and keeps place-bound times in local time, tools meant to vary vary, and its two judges stop failing what they shouldn't. Every change stays inside the probe.pet_8f3c1d92where the detail tool takes an integerpet_id), and every follow-up call then failed its schema check. Both now read the other offered tools with their parameter types, one per line (_other_endpoints), and an ID or other value a response returns has the type and form those endpoints take for it; the judge fails a response that returns one they don't take.The
tool_callingprompt version isv1.12.prompts.yamland the compiled defaults inprompts.pychange together.How it was verified
New tests in
tests/probes/tool_calling/test_probe.pyfail on the code before each change and pass after it: the other endpoints listed with their parameter types and the called tool left out, both prompts reading them throughexecute_tool_call, the local-time line, the line for random and generating tools, the response judge's line on fields describing the response, the per-turn judge's exemptions, and the opening check's new failure, absent from follow-ups.On every commit's own tree:
make check-all, the pre-commit hooks on the commit's files,pytest --collect-only(5237 to 5242 tests collected) and thetool_callingtests.At the head:
Live, against a custom provider config, on the head's code: the end-to-end health check with the
tool_callingprobe, in English, and onetool_callingrow simulated and evaluated with--scorers auto:Results impact
tool_callingconversations from prompt versionv1.11are regenerated atv1.12. Simulated responses return IDs that the other offered tools accept, so calls that failed their schema check on such an ID now go through; place-bound times read in local time; and random or generating tools no longer repeat themselves. The response judge rejects fewer responses over fields that describe the response, the per-turn judge no longer fails justified refusals or accurate statements that no offered tool does what is asked, and openings that can't be acted on yet are written again. Assistant scores can move in both directions.