Add handle-based Python C API for per-loop isolation#5
Merged
Conversation
Implement per-loop pending queues so each erlang_event_loop_t instance is fully isolated. This enables multiple independent event loops for sub-interpreter support and concurrent asyncio usage. New Python C API methods (_for variants): - _loop_new() -> capsule: Create new event loop - _loop_destroy(loop): Destroy event loop - _run_once_native_for(loop, timeout_ms): Run once on specific loop - _add_reader_for/_remove_reader_for: FD read monitoring - _add_writer_for/_remove_writer_for: FD write monitoring - _schedule_timer_for/_cancel_timer_for: Timer operations - _wakeup_for/_is_initialized_for/_get_pending_for: Utilities New Erlang NIFs: - reselect_reader_fd/1, reselect_writer_fd/1: FD-only reselect using fd_res->loop backref (simplifies router code) Python ErlangEventLoop updates: - Added _loop_handle slot for per-loop isolation - All methods use _for variants when handle is set - Backward compatible: legacy API still uses global loop Test suite: - New py_multi_loop_SUITE.erl with isolation tests - All 108 tests pass
Feature flag:
- Add {event_loop_isolation, global | per_loop} to app env
- Default is 'global' for backward compatibility
- When 'per_loop', each ErlangEventLoop creates its own isolated native loop
Implementation:
- Add g_isolation_mode global flag in C
- Add nif_set_isolation_mode/1 NIF to set the mode
- Add py_get_isolation_mode() Python function to query the mode
- Update ErlangEventLoop.__init__ to auto-create isolated loop in per_loop mode
- py_event_loop:init reads env and sets isolation mode on startup
Integration tests (py_multi_loop_integration_SUITE):
- test_per_loop_mode_enabled: verify mode is correctly set
- test_two_loops_concurrent_sleep: two threads with isolated loops
- test_two_loops_concurrent_gather: isolation with unique markers
- test_isolated_loop_tcp_echo: multiple loop create/destroy
All 112 tests pass.
Per-loop created loops (via _loop_new()) now automatically use a shared router, enabling full asyncio support including timers and FD operations. Implementation: - Add global shared router storage in C (g_shared_router) - Add set_shared_router/1 NIF to set the shared router - py_loop_new() automatically assigns shared router to new loops - Timer messages now include LoopRef for correct dispatch - Router stores LoopRef with each timer, dispatches to correct loop Test: - Add test_two_loops_with_timers verifying timer callbacks work - All 113 tests pass
benoitc
force-pushed
the
feature/per-loop-pending-queues
branch
from
e391ce9 to
e36c3f9
Compare
Instead of a global mode switch, users can now create isolated loops explicitly with ErlangEventLoop(isolated=True). This is simpler and more intuitive. Changes: - Add isolated=False parameter to ErlangEventLoop.__init__ - Remove event_loop_isolation app env setting - Remove set_isolation_mode call from py_event_loop init - Update integration tests to use isolated=True - Update docs/asyncio.md with new API - Update CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_formethods) enabling multiple independent event loopserlang_event_loop_tinstance is now fully isolated with its own pending queueChanges
New Python C API methods:
_loop_new()/_loop_destroy(loop)- lifecycle management_run_once_native_for(loop, timeout_ms)- run once on specific loop_add_reader_for/_remove_reader_for- FD read monitoring_add_writer_for/_remove_writer_for- FD write monitoring_schedule_timer_for/_cancel_timer_for- timer operations_wakeup_for/_is_initialized_for/_get_pending_for- utilitiesNew Erlang NIFs:
reselect_reader_fd/1,reselect_writer_fd/1- FD-only reselect usingfd_res->loopbackrefPython ErlangEventLoop:
_loop_handleslot for per-loop isolation_forvariants when handle is setTests:
py_multi_loop_SUITE.erlwith isolation tests