Is your feature request related to a problem?
mxcp test uses .yml tags to specify tests.
I'm working on a tool that runs a query returning a list of rows. Because that can be long, I'll wrap all this in an object with the number of rows, and a boolean saying if that was truncated. I believe the only thing I can do then is test top-level fields, I can't assert anything in the nested results, make sure there exist a row that has key X with value Y.
Describe the solution you'd like
A standard/simple way to implement tests in python and they would be run by mxcp test.
Describe alternatives you've considered
Add more keywords to the .yml test spec, maybe even something would work here, but it feels we'll eventually need to run arbitrary logic (ensure rows are in an order, etc.)
Use Case
It's a good practice to add metadata around tool results, like number of rows. That shouldn't impact the level of testing.
Implementation Ideas
A python library equivalent to mxcp run tool. Because the tool name would be known, mxcp test would run all tests, mxcp test tool X could walk the tests and skip those that aren't tool X? Also tool name is specified in .yml so one can really generate a hardcoded python module that matches the tools/prompts/resources.
def test_salesforce_user_found():
tool_result = tool.search_user(name=...)
assert ...
Is your feature request related to a problem?
mxcp test uses .yml tags to specify tests.
I'm working on a tool that runs a query returning a list of rows. Because that can be long, I'll wrap all this in an object with the number of rows, and a boolean saying if that was truncated. I believe the only thing I can do then is test top-level fields, I can't assert anything in the nested results, make sure there exist a row that has key X with value Y.
Describe the solution you'd like
A standard/simple way to implement tests in python and they would be run by mxcp test.
Describe alternatives you've considered
Add more keywords to the .yml test spec, maybe even something would work here, but it feels we'll eventually need to run arbitrary logic (ensure rows are in an order, etc.)
Use Case
It's a good practice to add metadata around tool results, like number of rows. That shouldn't impact the level of testing.
Implementation Ideas
A python library equivalent to
mxcp run tool. Because the tool name would be known,mxcp testwould run all tests,mxcp test tool Xcould walk the tests and skip those that aren't tool X? Also tool name is specified in.ymlso one can really generate a hardcoded python module that matches the tools/prompts/resources.