Test failure on Py3.11

[sanjayankur31]

A test is failing on Py3.11 here:

======================================================================
FAIL: test_source_check (sciunit.unit_test.model_tests.CapabilitiesTestCase.test_source_check)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/asinha/Documents/02_Code/01_others/sciunit/sciunit/unit_test/model_tests.py", line 241, in test_source_check
    self.assertFalse(MyCap1.source_check(MyModel2()))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: True is not false

----------------------------------------------------------------------
Ran 83 tests in 11.860s

FAILED (failures=1, skipped=2)

Steps to reproduce:

• clone repo
• install py3.11 and associated developer libraries/headers
• set up py3.11 virtualenv
• pip install wheel pytest
• pip install .
• chmod +x test.sh
• ./test.sh

[sanjayankur31]

Just double-checked: all tests pass on py3.10:

Ran 83 tests in 9.938s

OK (skipped=2)

[rgerkin]

Do you know why this fails in 3.11?

[sanjayankur31]

Not sure yet, didn't see anything obvious in the py3.11 release notes that may explain it, so will require a little bit of digging.

[ChihweiLHBird]

The reason was the behavior of Python dis module from the standard library has been changed in Python 3.11.

before 3.11	3.11

Which causes our source check failed

stdout = sys.stdout
sys.stdout = io.StringIO()

dis.dis(getattr(cls, method))

dis_output = sys.stdout.getvalue()
sys.stdout = stdout

dis_output = re.split("\n|\s+", dis_output)
dis_output = [word for word in dis_output if word]

if (
    "(NotImplementedError)" in dis_output
    or "(unimplemented)" in dis_output
    or "(CapabilityNotImplementedError)" in dis_output
    or "(NotImplemented)" in dis_output
):
    cap_source = inspect.getsource(getattr(cls, method))
    model_source = inspect.getsource(getattr(model, method))

    if cap_source == model_source:
        source_capable = False
        break

Because it is now (NULL + NotImplementedError) instead of (NotImplementedError), we can make an ambitious change to check whether NotImplementedError is the substring of the output, but this might cause trouble if there is some other disassembled normal code that includes these strings. This is not very likely, though. Another solution in my mind is to overhaul the Capability system to utilize Python Abstract Base Class to enforce the abstract methods’ implementation, but this is a relatively large change.