Add missing trace check for debug output in compile_plan

[mferencevic]

Motivation

After the last few changes on the develop branch we've started getting the following output in our logs that we couldn't remove with any environment variable modifications:

Skipped 4 configs for gpu::mlir_op

Technical Details

This is the only call to std::cout in the entire file (src/targets/gpu/compile_ops.cpp) that didn't check trace before calling std::cout. You may want me to tweak the trace level that is checked.

Changelog Category

Add a CHANGELOG.md entry for any option other than Not Applicable

• • Added: New functionality.
• • Changed: Changes to existing functionality.
• • Removed: Functionality or support that has been removed. (Compared to a previous release)
• • Optimized: Component performance that has been optimized or improved.
• • Resolved Issues: Known issues from a previous version that have been resolved.
• • Not Applicable: This PR is not to be included in the changelog.

[pfultz2]

This is supposed to be always printed.

[eddieliao]

LGTM

[pfultz2]

The print should not be removed here. This is not a trace message. These messages should not be ignored, the corresponding kernel needs to update its applicability check when you see the message. We keep the message concise to avoid spamming the user with the full error messages, but the visibility is important.

[pfultz2]

We can move these message to use the logger or use cerr if thats a better choice.

[mferencevic]

@pfultz2 Thanks for the comments. Although, I want to share that the logs as they are currently don't make any sense for us (a user of MIGraphX).

Basically, our logs that are well structured are currently being polluted with the following outputs:

Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 2 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 4 configs for gpu::mlir_op
Skipped 3 configs for gpu::mlir_op
Skipped 3 configs for gpu::mlir_op

As you've mentioned, this could be an important warning/error for you (as MIGraphX developers), but the logs as they are currently formatted don't have any information on which model triggered the issue (and we compile a lot of different models), don't have any information on what printed them (we had to dig through the MIGraphX codebase) and aren't suppressible in any way.

In my opinion if they indicate significant compilation errors they should be raised to an error level and should fail the compilation entirely - otherwise, they should be more verbose and somehow suppressible from our logs because we don't want our end users seeing them. I don't think that using std:cerr would be a good option because again, they would show up in our error logs.

I don't have any preference on which option makes most sense, what I've implemented here seemed most logical to me as the trace variable is already used for similarly formatted and looking messages.

[pfultz2]

We can change it to use log::warn() or log::info() and you can set MIGRAPHX_LOG_LEVEL=0 to suppress it. We can also expose the log:::set_severity function to the API so this can be handled at the API level as well.

[mferencevic]

I don't have any preference there, log::warn and log::info sound like good options to me.

Feel free and decide what it should be changed to.