test: Add tests for intrinsics-related formatters

[frreiss]

Misc PR

Type of PR

• Bug Fix
• New Feature
• Documentation
• Other

Description

This PR adds additional tests for the intrinsics-related formatting code added in #571. I've also added a dependency on the pytest-recording plugin that several of these tests use.

[github-actions]

The PR description has been updated. Please fill out the template for your PR to be reviewed.

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert|release)(?:\(.+\))?:

[frreiss]

CI appears to be up to its usual shenanigans. Tests are passing on my machine.

[jakelorocco]

Thanks @frreiss; these changes / tests look reasonable. I will try to investigate the CI failures soon. I don't want to merge and break the checks on other PRs if possible.

[jakelorocco]

I will continue investigating, but I believe this is probably due to memory and/or storage constraints. If I'm reading the tests correctly, the test is failing on test_run_transformers which is the first test that starts loading new models / downloading adapters.

[frreiss]

CI passing now.

[jakelorocco]

@frreiss, are you certain it was a timeout issue? I'm seeing the failed jobs getting cancelled after 8-13minutes.

[frreiss]

@frreiss, are you certain it was a timeout issue? I'm seeing the failed jobs getting cancelled after 8-13minutes.

The error condition is consistent with a timeout. Here's an example stack trace:

test/formatters/granite/test_intrinsics_formatters.py::test_reparse_json[bare_string.json] PASSED [ 51%]
##[debug]Re-evaluate condition on job cancellation for step: 'Run Tests'.
test/formatters/granite/test_intrinsics_formatters.py::test_run_transformers[answerability_simple] 
Error: The operation was canceled.
##[debug]System.OperationCanceledException: The operation was canceled.
##[debug]   at System.Threading.CancellationToken.ThrowOperationCanceledException()
##[debug]   at GitHub.Runner.Sdk.ProcessInvoker.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Channel`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, CancellationToken cancellationToken)
##[debug]   at GitHub.Runner.Common.ProcessInvokerWrapper.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Channel`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, CancellationToken cancellationToken)
##[debug]   at GitHub.Runner.Worker.Handlers.DefaultStepHost.ExecuteAsync(IExecutionContext context, String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Boolean inheritConsoleHandler, String standardInInput, CancellationToken cancellationToken)
##[debug]   at GitHub.Runner.Worker.Handlers.ScriptHandler.RunAsync(ActionRunStage stage)
##[debug]   at GitHub.Runner.Worker.ActionRunner.RunAsync()
##[debug]   at GitHub.Runner.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
##[debug]Finishing: Run Tests

(from the build at https://github.com/generative-computing/mellea/actions/runs/22880035073/job/66523072098#step:13:307).

From the stack trace, it looks like something is sending a cancellation signal inside the Javascript code that runs the workflow.

The Github records for these builds tend to show a time that is less than 15 minutes (13:31 in the case of the above build), but actual wall clock time from the workflow starting until the first failed test run is very close to 15 minutes.

Many time-consuming tests are disabled when running in CI mode; see https://github.com/search?q=repo%3Agenerative-computing%2Fmellea%20gh_run&type=code

[jakelorocco]

Okay, confirmed it's a timeout issue by testing with an extended timeout here: #632.

I will approve / merge these changes as is and will discuss with the larger team about extending the timeout / winnowing down our tests.

Thank you!