Trigger OOM at the end of GCs

[qinsoon]

Currently the only place we throw OOM is during allocation. But we allow APIs to count the memory allocated by bindings/runtimes as heap memory. So in rare cases where runtimes keep allocating memory and count them as heap memory, but do not use MMTk allocation, we will see heap size exceed the heap limit, but no OOM is triggered.

We could consider add an OOM check at the end of GCs.

[wks]

The VM can, in theory, allocate off-heap memory in any form (e.g. malloc, mmap, etc.) and count it as part of Collections::vm_live_bytes(). Because this is highly VM-specific, the VM can do such allocations at safepoints or not at safepoints. When not at safepoint, it will overcommit and have to postpone polling and GC to a latter point. This is very similar to the use cases we are currently handling using AllocationOptions, but it is for native allocation.

Currently, our MMTk-binding API provides two functions for the binding to trigger GC.

• MMTK::handle_user_collection_request
• memory_manager::gc_poll

The former triggers GC unconditionally and block for GC. The latter does polling and, if GC is triggered, block for GC.

If a VM binding always does native allocation at safepoints, the VM binding only needs to call memory_manager::gc_poll at every such allocation site, or every time it has allocated a certain number of pages.

What if VM bindings can allocate off-heap memory outside safepoint?

Currently there is no way to do the polling while not at safepoint. One way to fix this is to introduce another API function:

struct PollingOptions {
    at_safepoint: bool,
}

impl MMTK<VM>
    fn gc_poll(&self, options: &PollingOptions);
}

If at_safepoint is true, it behaves the same as the current memory_manager::gc_poll. We can make memory_manager::gc_poll a wrapper of mmtk.gc_poll(&PollingOptions { at_safepoint: true}).

If at_safepoint is false, it will trigger GC, but not block for GC. GC will happen later if

• a thread performs GC allocation at safepoint (Allocator::alloc), or
• a thread enters safepoint and performs an explicit MMTk::gc_poll with at_safepoint: true.

If a VM frequently does allocation outside safepoint, it should compensate this by entering safepoint from time to time and poll.