Improve assertions error messages with structured format by Evangelink · Pull Request #7444 · microsoft/testfx

Evangelink · 2026-02-20T21:33:12Z

Copilot

Pull request overview

This pull request modernizes assertion error messages by introducing a structured, multi-line format that improves readability and developer experience. The changes replace old-style concatenated messages with formatted parameter displays using raw string literals.

Changes:

Introduced structured error message formatting with parameter names, expressions, and values on separate lines
Added helper methods for value formatting, expression truncation, and redundancy detection
Updated all assertion methods to use the new formatting approach
Removed obsolete resource strings and added new simplified ones
Updated all test expectations to match the new message format

Reviewed changes

Copilot reviewed 41 out of 41 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
Assert.cs	Core formatting infrastructure: FormatValue, FormatParameter, truncation logic
Assert.ThrowsException.cs	Updated to new structured format for exception type mismatches
Assert.StartsWith/EndsWith/Matches.cs	String assertion methods using new format
Assert.IsTrue/IsFalse.cs	Boolean assertion methods with condition parameter display
Assert.IsNull/IsNotNull.cs	Null checking assertions with value display
Assert.IsInstanceOfType/IsExactInstanceOfType.cs	Type checking with structured type comparison
Assert.IComparable.cs	Comparison assertions (greater/less than, positive/negative)
Assert.Count/Contains.cs	Collection assertions with expression parameters
Assert.AreSame.cs	Reference equality with hash code display for disambiguation
FrameworkMessages.resx	Resource string simplification and new message keys
xlf files	Localization files marked with untranslated entries
Test files	Updated expectations for all assertion error messages

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.IsInstanceOfTypeTests.cs

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.IsExactInstanceOfTypeTests.cs

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.AreSame.cs

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.IsExactInstanceOfTypeTests.cs

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.Items.cs

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.IsTrueTests.cs

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.IsNull.cs

Copilot

Pull request overview

Copilot reviewed 44 out of 44 changed files in this pull request and generated 7 comments.

src/TestFramework/TestFramework/Assertions/Assert.cs

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.cs

src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.zh-Hans.xlf

src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.pt-BR.xlf

src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.es.xlf

…r messages

Copilot

Pull request overview

Copilot reviewed 44 out of 44 changed files in this pull request and generated no new comments.

nohwnd · 2026-02-23T11:48:04Z

looks great

Copilot

Pull request overview

Copilot reviewed 46 out of 46 changed files in this pull request and generated 4 comments.

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.cs

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.StartsWith.cs

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.MatchesRegex.cs

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.EndsWith.cs

src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs

Youssef1313 · 2026-03-10T11:37:37Z

src/TestFramework/TestFramework/Assertions/Assert.AreSame.cs

+        string actualFormatted = FormatValue(actual);
+        bool sameToString = expected is not null && actual is not null && expectedFormatted == actualFormatted;
+        string expectedValue = sameToString
+            ? expectedFormatted + $" (HashCode={RuntimeHelpers.GetHashCode(expected!)})"


This might be confusing to users as-is. Instead of HashCode=, maybe we should be explicit this is RuntimeHelpers.GetHashCode. Otherwise, users might think this is equivalent to x.GetHashCode() (the difference is only if the type overrides the default GetHashCode)

Maybe just Hash=XXX?

src/TestFramework/TestFramework/Assertions/Assert.cs

Youssef1313 · 2026-03-10T11:40:14Z

src/TestFramework/TestFramework/Assertions/Assert.cs

        {
-            string finalMessage = string.Format(CultureInfo.CurrentCulture, FrameworkMessages.NullParameterToAssert, parameterName);
-            ThrowAssertFailed(assertionName, finalMessage);
+            foreach (object? item in collection)


Multiple enumerations are extremely risky IMO.
The original assert API that failed has likely already started enumerating the collection.

There is no guarantee that enumerating the given IEnumerable will be leading to the same original result as the first enumeration that happened earlier.

Youssef1313 · 2026-03-10T11:42:17Z

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.AreSame.cs

+        action.Should().Throw<Exception>().WithMessage("""
+            Assert.AreSame failed.
+            Expected references to be the same.
+              expected (new object()): <System.Object> (HashCode=*)


Should this look like the following instead?

expected 'new object()': <System.Object> (HashCode=*)

Yeah probably (even though then it looks like string...), personally I always trip over the expressions in the assertion messages lately, it is hard to know what is going on there when your code that produces the value is longer than few characters.

or when your expression is actual, more examples on the general approach would be nice that explore what will happen for common usecases.

Commented in few other places, where this seems problematic.

nohwnd · 2026-03-10T13:28:15Z

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.AreEqualTests.cs

+            .WithMessage("""
+                Assert.AreEqual failed.
+                Expected values to be equal.
+                  expected (new object()): <System.Object> (System.Object)


Looking through the code to see places where I would be confused.

This is imho very hard to interpret. The additional info about expressions and types should aid in understanding what is going on. And I would avoid putting anything before the actual values, instead I would prefer aligning the values on the same level, and add additional information after if needed, something like:

Assert.AreEqual failed. Expected values to be equal. expected: System.Object [System.Object] actual: 1 [System.Int32] at Assert.AreEqual(new object(), 1).

This seems okay, and IMHO I have all the info needed. The values, the types, and what was called.

What does the < > do in the output? Is that telling me that this was unprintable value or something?

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.AreEqualTests.cs

nohwnd · 2026-03-10T13:37:22Z

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.AreEqualTests.cs

+            .WithMessage("""
+                Assert.AreNotEqual failed. User-provided message. DummyClassTrackingToStringCalls,     DummyClassTrackingToStringCalls, Hello, DummyIFormattable.ToString()*
+                Expected a difference greater than <0.2>.
+                  notExpected: 1.0m


This is existing code, but I am not 100% on board with this naming, makes me do additional mental math on what you meant here. The message does great job at telling me what is the expectation, and it already uses language appropriate for the Not*Assertion.

negative assertion:
Expected a difference greater than <0.2>.

positive assertion:
Expected a difference no greater than <0>.

the <> also seem extra, do they serve some purpose somewhere?

I don't get the first comment, what's the problem and what's your suggestion?

Expected a difference greater than <0.2>. notExpected: 1.0m actual: 1.0m
Expected a difference no greater than <0>. expected: 1.0m actual: 0.9m

1.0m is the value I provided, that drives the decision if the assertion will fail or not, but it says "notExpected/unexpected".

nohwnd

You might find it helpful to look at how pester does the formatting, this PR has lot of problems (this is subjective) that are solved there:
https://github.com/pester/Pester/blob/d5f7c87a5fd2409f77eb73bd891e7f15a3eb6a80/src/Format2.ps1#L4

nohwnd · 2026-03-10T13:39:35Z

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.AreEqualTests.cs

-            -------------^
+            Assert.AreEqual failed.
+            String lengths are both 4 but differ at index 2.
+              expected ("aa\ta"): "aa?a"


One more example of the expression not really helping to explain what is going on

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.AreEqualTests.cs

nohwnd · 2026-03-10T13:40:46Z

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.AreEqualTests.cs

+            Assert.AreEqual failed.
+            String lengths are both 201 but differ at index 100.
+              expected: "...aaaaaaaaaaaaaaaaaabcccccccccccccccc..."
+              actual:   "...aaaaaaaaaaaaaaaaaadcccccccccccccccc..."


values are aligned, this is how it should be everywhere.

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.AreEqualTests.cs

nohwnd · 2026-03-10T13:42:51Z

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.AreSame.cs

+        action.Should().Throw<Exception>().WithMessage("""
+            Assert.AreSame failed.
+            Expected references to be the same.
+              expected (new object()): <System.Object> (HashCode=*)


Commented in few other places, where this seems problematic.

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.cs

nohwnd · 2026-03-10T13:54:59Z

That said this is great effort 👍 I love more consistency for the assertion messages!

Copilot

Pull request overview

Copilot reviewed 46 out of 46 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (2)

src/TestFramework/TestFramework/Assertions/Assert.IsInstanceOfType.cs:343

When expectedType is null (one of the documented failure cases), the failure message currently only prints the value line and omits any indication that expectedType was null. This makes the assertion failure hard to diagnose. Consider either validating expectedType with CheckParameterNotNull (like other asserts do) or including an explicit "expectedType: (null)" / reason line in the structured output.

        string message = string.IsNullOrEmpty(userMessage) ? string.Empty : userMessage!;
        if (expectedType is not null && value is not null)
        {
            message += Environment.NewLine + FrameworkMessages.IsInstanceOfTypeFailNew;
        }

        message += Environment.NewLine + FormatParameter(nameof(value), valueExpression, value);
        if (expectedType is not null && value is not null)
        {
            message += Environment.NewLine + $"  expectedType: {expectedType}"
                     + Environment.NewLine + $"  actualType: {value.GetType()}";
        }

src/TestFramework/TestFramework/Assertions/Assert.IsExactInstanceOfType.cs:336

When expectedType is null, IsExactInstanceOfType fails but the structured message doesn't indicate that expectedType was null (it only prints the value line). Consider validating expectedType with CheckParameterNotNull or adding an explicit "expectedType: (null)" / reason line so callers can understand why the assert failed.

    private static void ThrowAssertIsExactInstanceOfTypeFailed(object? value, Type? expectedType, string? userMessage, string valueExpression)
    {
        string message = string.IsNullOrEmpty(userMessage) ? string.Empty : userMessage!;
        if (expectedType is not null && value is not null)
        {
            message += Environment.NewLine + FrameworkMessages.IsExactInstanceOfTypeFailNew;
        }

        message += Environment.NewLine + FormatParameter(nameof(value), valueExpression, value);
        if (expectedType is not null && value is not null)
        {
            message += Environment.NewLine + $"  expectedType: {expectedType}"
                     + Environment.NewLine + $"  actualType: {value.GetType()}";
        }

Copilot · 2026-03-11T15:22:48Z

src/TestFramework/TestFramework/Assertions/Assert.IsInstanceOfType.cs

+        if (wrongType is not null)
+        {
+            message += Environment.NewLine + FrameworkMessages.IsNotInstanceOfTypeFailNew;
+        }
+
+        message += Environment.NewLine + FormatParameter(nameof(value), valueExpression, value);
        if (wrongType is not null)
        {


When wrongType is null, IsNotInstanceOfType fails but the structured message doesn't mention that wrongType was null (it only prints the value line). Since wrongType being null is a key diagnostic, consider validating wrongType with CheckParameterNotNull or including an explicit "wrongType: (null)" / reason line in the output.

This issue also appears on line 332 of the same file.

Suggested change

if (wrongType is not null)

{

message += Environment.NewLine + FrameworkMessages.IsNotInstanceOfTypeFailNew;

}

message += Environment.NewLine + FormatParameter(nameof(value), valueExpression, value);

if (wrongType is not null)

{

message += Environment.NewLine + FrameworkMessages.IsNotInstanceOfTypeFailNew;

message += Environment.NewLine + FormatParameter(nameof(value), valueExpression, value);

if (wrongType is null)

{

message += Environment.NewLine + " wrongType: (null)";

}

else

{

Copilot · 2026-03-11T15:22:48Z

src/TestFramework/TestFramework/Assertions/Assert.IsExactInstanceOfType.cs

-                value!.GetType().ToString());
+            message += Environment.NewLine + $"  wrongType: {wrongType}"
+                     + Environment.NewLine + $"  actualType: {value!.GetType()}";
        }


When wrongType is null, IsNotExactInstanceOfType fails but the structured message omits any mention that wrongType was null. Since wrongType being null is a primary cause, consider validating wrongType with CheckParameterNotNull or including a "wrongType: (null)" / reason line in the message.

This issue also appears on line 323 of the same file.

Suggested change

}

}

else

{

message += Environment.NewLine + " wrongType: (null)"

+ Environment.NewLine + " reason: The parameter 'wrongType' must not be null.";

}

Copilot · 2026-03-11T15:22:48Z

src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.zh-Hans.xlf

+      <trans-unit id="AreEqualFailNew">
+        <source>Expected values to be equal.</source>
+        <target state="new">Expected values to be equal.</target>
+        <note />


Per repo localization guidelines, *.xlf files shouldn't be edited manually. These XLF changes should be produced by updating the .resx and then building to regenerate the XLFs. Please revert any hand edits and regenerate the XLF file(s) from the build pipeline to avoid translation merge issues.

Suggested change

<note />

<note></note>

Copilot · 2026-03-11T15:22:49Z

src/TestFramework/TestFramework/Assertions/Assert.cs

+        // the format produces a trailing space before the newline ("failed. \\r\\n").
+        // Remove it to avoid trailing whitespace on the first line.
+        if (message is not null && message.StartsWith(Environment.NewLine, StringComparison.Ordinal))
+        {
+            formattedMessage = $"{assertionName} failed.{message}";


ThrowAssertFailed special-cases messages that start with a newline by constructing the prefix with a hardcoded English string ("{assertionName} failed."). This bypasses the localized FrameworkMessages.AssertionFailed format and will regress non-English assertion prefixes whenever structured messages start with a newline. Consider keeping the localization by deriving the prefix from FrameworkMessages.AssertionFailed (e.g., formatting with an empty message and trimming trailing whitespace) rather than hardcoding "failed.".

Suggested change

// the format produces a trailing space before the newline ("failed. \\r\\n").

// Remove it to avoid trailing whitespace on the first line.

if (message is not null && message.StartsWith(Environment.NewLine, StringComparison.Ordinal))

{

formattedMessage = $"{assertionName} failed.{message}";

// the format produces a trailing space before the newline (for example, "failed. \\r\\n").

// Remove it to avoid trailing whitespace on the first line while preserving localization.

if (message is not null && message.StartsWith(Environment.NewLine, StringComparison.Ordinal))

{

string prefix = string.Format(CultureInfo.CurrentCulture, FrameworkMessages.AssertionFailed, assertionName, string.Empty);

formattedMessage = prefix.TrimEnd() + message;

Copilot · 2026-03-11T15:22:49Z

src/TestFramework/TestFramework/Assertions/Assert.cs

+        Type type = typeof(T);
+        if (type == typeof(object))
        {
-            return userMessage;
+            // If the static type is object, use the runtime type
+            type = value.GetType();
        }

-        string callerArgMessagePart = string.Format(CultureInfo.InvariantCulture, FrameworkMessages.CallerArgumentExpressionTwoParametersMessage, parameterName1, callerArgExpression1, parameterName2, callerArgExpression2);
-        return string.IsNullOrEmpty(userMessage)
-            ? callerArgMessagePart
-            : $"{callerArgMessagePart} {userMessage}";
+        if (type.IsPrimitive || value is decimal or DateTime or DateTimeOffset
+            or TimeSpan or Guid or Enum)
+        {
+            return EscapeNewlines(Truncate(value.ToString() ?? string.Empty, maxLength));
+        }
+
+        MethodInfo? toStringMethod = type.GetMethod(nameof(ToString), BindingFlags.Public | BindingFlags.Instance, null, Type.EmptyTypes, null);
+        if (toStringMethod is not null
+            && toStringMethod.DeclaringType != typeof(object)
+            && toStringMethod.DeclaringType != typeof(ValueType))
+        {
+            try
+            {
+                return EscapeNewlines(Truncate(value.ToString() ?? string.Empty, maxLength));
+            }
+            catch (Exception)
+            {
+                // Fall through to type name display if ToString throws
+            }
+        }
+
+        // No useful ToString - just return the type name
+        string typeName = type.FullName ?? type.Name;
+        return $"<{typeName}>";


FormatValue uses typeof(T) (and only switches to the runtime type when T==object). For values typed as an interface/base class where the runtime type overrides ToString(), this can incorrectly fall back to displaying the static type name (e.g., "") instead of the runtime ToString() output. Consider basing the ToString/Type checks on value.GetType() for non-null values (while still keeping the existing primitive/collection handling).

Copilot · 2026-03-11T15:22:49Z

src/TestFramework/TestFramework/Resources/xlf/FrameworkMessages.es.xlf

+      <trans-unit id="AreEqualFailNew">
+        <source>Expected values to be equal.</source>
+        <target state="new">Expected values to be equal.</target>
+        <note />


Per repo localization guidelines, *.xlf files shouldn't be edited manually. These XLF changes should be produced by updating the .resx and then building to regenerate the XLFs. Please revert any hand edits and regenerate the XLF file(s) from the build pipeline to avoid translation merge issues.

Suggested change

<note />

<note></note>

Assert.AreEqual("aaaa", "aaab") failed. String lengths are both 4 but differ at index 3. expected: "aaaa" actual: "aaab" --------------^

Copilot AI review requested due to automatic review settings February 20, 2026 21:33

Copilot started reviewing on behalf of Evangelink February 20, 2026 21:33 View session

Copilot AI reviewed Feb 20, 2026

View reviewed changes

Add assertion formatting helpers and localized resource strings

8014b3c

Evangelink force-pushed the dev/amauryleve/rework-assert branch 2 times, most recently from a55c762 to 49018c6 Compare February 22, 2026 10:31

Copilot AI review requested due to automatic review settings February 22, 2026 10:31

Copilot started reviewing on behalf of Evangelink February 22, 2026 10:31 View session

Copilot AI reviewed Feb 22, 2026

View reviewed changes

Evangelink added 14 commits February 22, 2026 11:37

Improve AreEqual/AreNotEqual assertion error messages

b2127af

Improve AreSame/AreNotSame assertion error messages

9ac4849

Improve IsTrue/IsFalse assertion error messages

375f0b5

Improve IsNull/IsNotNull assertion error messages

bd66356

Improve IsInstanceOfType/IsNotInstanceOfType assertion error messages

7e7f07e

Improve IsExactInstanceOfType/IsNotExactInstanceOfType assertion erro…

bebe2ef

…r messages

Improve IComparable assertion error messages

d35c33b

Improve HasCount/IsEmpty/IsNotEmpty assertion error messages

23758d2

Improve ThrowsException assertion error messages

3705c74

Improve Contains/DoesNotContain/IsInRange assertion error messages

23cd627

Improve StartsWith/DoesNotStartWith assertion error messages

dd88189

Improve EndsWith/DoesNotEndWith assertion error messages

0c10f63

Improve MatchesRegex/DoesNotMatchRegex assertion error messages

14a6ec3

Minor test cleanup for Inconclusive assertion

63e8257

Evangelink force-pushed the dev/amauryleve/rework-assert branch from 49018c6 to 63e8257 Compare February 22, 2026 10:38

Address Copilot feedback

f559387

Copilot AI review requested due to automatic review settings February 22, 2026 12:56

Copilot started reviewing on behalf of Evangelink February 22, 2026 12:56 View session

Copilot AI reviewed Feb 22, 2026

View reviewed changes

Fix acceptance tests

dfe9891

Copilot started reviewing on behalf of Evangelink February 23, 2026 14:06 View session

Copilot AI reviewed Feb 23, 2026

View reviewed changes

Evangelink enabled auto-merge March 2, 2026 15:28

Evangelink mentioned this pull request Mar 10, 2026

Proposal: Assert.AreSequenceEqual<T>(IEnumerable<T> actual, IEnumerable<T> expected) #7503

Open

Youssef1313 self-requested a review March 10, 2026 11:19

Youssef1313 self-assigned this Mar 10, 2026

Youssef1313 reviewed Mar 10, 2026

View reviewed changes

src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs Outdated Show resolved Hide resolved

Youssef1313 reviewed Mar 10, 2026

View reviewed changes

src/TestFramework/TestFramework/Assertions/Assert.cs Outdated Show resolved Hide resolved

Youssef1313 reviewed Mar 10, 2026

View reviewed changes

nohwnd reviewed Mar 10, 2026

View reviewed changes

test/UnitTests/TestFramework.UnitTests/Assertions/AssertTests.AreEqualTests.cs Outdated Show resolved Hide resolved

nohwnd reviewed Mar 10, 2026

View reviewed changes

Evangelink added 8 commits March 11, 2026 09:14

Merge branch 'main' into dev/amauryleve/rework-assert

4422c82

Fix diff caret position

f725f60

Use X more chars instead of total length

9c96c9b

Use item instead of element

632c3c1

Use more items

fa263f2

Updates

2822182

Simplify user message handling

39b59eb

Update hash text

60d17a9

Copilot AI review requested due to automatic review settings March 11, 2026 15:15

Copilot started reviewing on behalf of Evangelink March 11, 2026 15:15 View session

Copilot AI reviewed Mar 11, 2026

View reviewed changes

Evangelink added 3 commits March 11, 2026 22:19

Improve

44c6ca1

Move expressions to first line

2610e3a

Assert.AreEqual("aaaa", "aaab") failed. String lengths are both 4 but differ at index 3. expected: "aaaa" actual: "aaab" --------------^

Avoid using wildcard in expected message

a402e6d

-        }
+        }
+        else
+        {
+            message += Environment.NewLine + "  wrongType: (null)"
+                     + Environment.NewLine + "  reason: The parameter 'wrongType' must not be null.";
+        }

Conversation

Evangelink commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

nohwnd commented Feb 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nohwnd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Evangelink commented Feb 20, 2026 •

edited

Loading