Implement event loop factory hook

[tjkuson]

Added the pytest_asyncio_loop_factories hook to parametrize asyncio tests with custom event loop factories. The hook can inspect item in order to configure different event loops for different tests.

import asyncio

import pytest
import uvloop


def pytest_asyncio_loop_factories(
    pytest.Config: config,
    pytest.Item: item
) -> Iterable[Callable[[], AbstractEventLoop]]:
    if item.get_closest_marker("uvloop"):
        return [uvloop.new_event_loop]
    else:
        return [asyncio.new_event_loop]

Closes #1101, #1032, #1346.

Relates to #1164 by building on the idea of having a global parametrization for all tests and fixtures instead of a marker. An alternative idea was to expose a configuration option where a user could describe event loop factors and which tests they applied to, but this seemed less ergonomic to me compared to the hook approach (and less powerful than allowing user-defined logic).

Test plan

Added new tests that pass via uvx tox.

Existing tests pass without changes.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.94%. Comparing base (dbacf7b) to head (d3c835d).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1373      +/-   ##
==========================================
+ Coverage   93.64%   94.94%   +1.30%     
==========================================
  Files           2        2              
  Lines         409      455      +46     
  Branches       44       53       +9     
==========================================
+ Hits          383      432      +49     
+ Misses         20       17       -3     
  Partials        6        6              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[seifertm]

Thanks for the great work @tjkuson ! This is much more than a draft. It looks like we finally have a replacement for the policy fixture in pytest-asyncio.

I did a couple of comments, but most of them are very minor. The two largest being the limitation to a single hook implementation and your thought about my idea to use the asyncio marker to limit the loop factory parametrization for a single test.

[seifertm]

I would prefer this to be prefixed with an underscore just to indicate in the fixture name that it's for internal use only.

[seifertm]

There are some test suites that use other plugins for async tests besides pytest-asyncio. Since pytest-trio/pytest-anyio and pytest-asyncio can coexist peacefully, the phrasing "each asynchronous test is run once per factory" made me wonder if this also affects async tests from other plugins.

The answer should be obvious, but to clear out any doubt, we could say "each pytest-asyncio test is run once per factory". Or we add another sentence that this doesn't affect tests not managed by pytest-asyncio.

[seifertm]

Can you share the reasoning behind not allowing multiple hook implementations?

How does pytest handle this for its native hooks?

[seifertm]

Since _collect_hook_loop_factories is run for virtually every test function, the following checks are also executed every time. My understanding is that this happens during collection time, because the exact set of test items isn't known before pytest_generate_tests finishes.

I don't think we need to take immediate action here. I merely want to point out that we should keep an eye on whether this has any significant impact on test collection performance. This can make life harder for users with large test suites.

[seifertm]

The content of this document is great. I just think it should be split over several documents to be consistent with the existing structutre.

Pytest-asyncio roughly follows the Diataxis documentation framework. I suggest to stop the How-to for custom loop factories right here at line 20, move 25–38 into a dedicated how-to guide, and pack the rest of the information into the reference section of the docs.

[seifertm]

Let's say a configured configured pytest_asyncio_loop_factories so that all tests to run with uvloop and SelectorEventLoop. There's one specific test that should only be run with uvloop. This means the user would have to switch to the conftest.py file and find some way to identify the test (e.g. custom marker or item name) to tell pytest-asyncio to use a different set of factories for this test.

Ergonomically, it would be preferable for the user to perform this change directly at the level of the test without having to change the hook implementation. This would also improve visibility that this test is an exception.

Based on the implementation of _get_item_loop_scope my understanding is that we have access to the asyncio marker here. Could we use the information from the marker to modify the test parametrization?

I'm thinking of something like this:

def pytest_asyncio_loop_factories(config, item):
    return {
        "default": asyncio.new_event_loop,
        "uvloop": uvloop.new_event_loop,
    }

async def parametrized_over_all_factories():
    ...

@pytest.mark.asyncio(loop_factories=["uvloop"])
async def  only_runs_with_uvloop():
    ...

@pytest.mark.asyncio(loop_factories=["default"])
async def  only_runs_with_default_asyncio_loop():
    ...

Do you think this is feasible and/or reasonable? I'd love to hear your opinion on this @tjkuson .