diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index 0f62cdecb..5e326538c 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md @@ -68,6 +68,7 @@ - [Event, Timer, and Task Priority](dxe_core/events.md) - [Image Loading and Execution](dxe_core/images.md) - [Memory Management](dxe_core/memory_management.md) + - [Memory Bins](dxe_core/memory_bins.md) - [Protocol Database](dxe_core/protocol_database.md) - [Synchronization](dxe_core/synchronization.md) - [Testing](dxe_core/testing.md) diff --git a/docs/src/dxe_core/memory_bins.md b/docs/src/dxe_core/memory_bins.md new file mode 100644 index 000000000..8a46deca0 --- /dev/null +++ b/docs/src/dxe_core/memory_bins.md @@ -0,0 +1,309 @@ +# Memory Bins + +Memory bins are pre-allocated regions of memory for specific EFI memory types that stabilize the UEFI runtime +memory footprint across boots. This stability is required for S4 (hibernate) resume, where the OS must restore +system memory to the same layout as the previous boot. Without bins, small variations in runtime memory allocation +patterns between boots can shift memory map entries and break S4 resume. + +For background on the S4 problem and the overall bin design, refer to the +[edk2 Memory Bin Feature](https://github.com/tianocore/edk2/blob/HEAD/MdeModulePkg/Core/MemoryBins.md) document. This +page focuses on the Patina implementation of the DXE side of that design. + +## How Bins Work + +A platform declares the desired bin sizes by producing a **Memory Type Information GUID HOB** +(`gEfiMemoryTypeInformationGuid`). Each entry in this HOB specifies a memory type and a page count representing +the bin size for that type. The memory map must not change during S4 resume because the OS will restore system memory +from disk. The memory bins keep memory ranges consistent for ranges of memory types that need to be consistent across +hibernation. + +- `EfiReservedMemoryType` + - Memory that is reserved for firmware use and may not be used by the OS. +- `EfiRuntimeServicesCode` + - Used for UEFI runtime services code sections. +- `EfiRuntimeServicesData` + - Used for UEFI runtime services data buffers. +- `EfiACPIMemoryNVS` + - Memory used or reserved by the system (e.g. ACPI FACS) and must not be used by the operating system. This memory is + required to be saved and restored across across an NVS sleep and saved in S4. +- `EfiACPIReclaimMemory` + - Used for most ACPI tables. Memory that is preserved by the loader and OS until ACPI is enabled and the ACPI tables + are read. This memory is required to be saved and restored across an NVS sleep cycle and saved in S4. + +When the Patina DXE Core initializes memory services, it: + +1. Finds the Memory Type Information HOB and extracts the bin configuration. +2. Establishes a contiguous address range for the bins. +3. Steers page allocations for bin types into their designated range. +4. Tracks allocation statistics so BDS can recommend next-boot bin sizes. +5. Adjusts `GetMemoryMap()` output so each bin appears as a single descriptor of its type, absorbing any free + space within the bin range. +6. Publishes the memory type information config table so BDS can recommend next-boot bin sizes. + +The bin size is an intentional over-allocation. Runtime allocations that fluctuate between boots are absorbed by +the bin, so the overall memory map reported to the OS remains stable. + +## Patina Implementation + +The implementation primarily resides in two files: + +- [`allocator.rs`](https://github.com/OpenDevicePartnership/patina/blob/HEAD/patina_dxe_core/src/allocator.rs) - + Integration points in the DXE Core memory allocator. +- [`memory_bin.rs`](https://github.com/OpenDevicePartnership/patina/blob/HEAD/patina_dxe_core/src/memory_bin.rs) - + The standalone `MemoryBinManager` module. + +### Input HOBs + +The bin feature consumes up to three types of HOBs from the PEI phase. The first HOB enables basic memory bin support +while the additional HOBs provide extra control over bin placement from the HOB producer phase. + +#### 1. Memory Type Information GUID HOB (required) + +A GUID Extension HOB (`EFI_HOB_TYPE_GUID_EXTENSION`) whose `Name` matches `gEfiMemoryTypeInformationGuid` +(`4C19049F-4137-4DD3-9C10-8B97A83FFDFA`). The HOB data is an array of `EFI_MEMORY_TYPE_INFORMATION` +entries terminated by a sentinel entry with `Type = EfiMaxMemoryType`: + +```text ++--------+--------+--------+--------+--------+--------+--- +| Type₀ | Pages₀ | Type₁ | Pages₁ | ... | 0x10 | 0 ++--------+--------+--------+--------+--------+--------+--- + u32 u32 u32 u32 sentinel +``` + +Each entry specifies a memory type and the number of 4 KB pages to reserve for that type's bin. + +**Fields used:** + +| Field | Size | Description | +|-----------------|-------|-------------------------------------------------------------------------------------------------------------------------------------------------| +| `Type` | `u32` | `efi::MemoryType` value (e.g. `EfiRuntimeServicesData = 6`). | +| `NumberOfPages` | `u32` | Bin size in 4 KB pages. The actual GCD allocation is aligned up to the type's granularity (64 KB for runtime types on AArch64, 4 KB otherwise). | + +If this HOB is absent, no bins are initialized and the feature is disabled. + +#### 2. Resource Descriptor HOB with Owner GUID (optional, recommended) + +A Resource Descriptor HOB (`EFI_HOB_TYPE_RESOURCE_DESCRIPTOR`) whose `Owner` field matches +`gEfiMemoryTypeInformationGuid`. This HOB describes a pre-allocated contiguous memory region that +PEI set aside for the bins. + +```text ++----------+--------------------------+------+------+-----------+-----------+ +| Header | Owner | Type | Attr | PhysStart | Length | +| (Hob) | 4C19049F-...-A83FFDFA | 0x00 | 0x07 | | | ++----------+--------------------------+------+------+-----------+-----------+ + ^ ^ + | PRESENT | INITIALIZED | TESTED + EFI_RESOURCE_SYSTEM_MEMORY +``` + +**Fields used:** + +| Field | Value/Constraint | Description | +|-------|------------------|-------------| +| `Owner` | `gEfiMemoryTypeInformationGuid` | Identifies this as the bin region HOB. | +| `ResourceType` | `EFI_RESOURCE_SYSTEM_MEMORY` (0) | Must be system memory. | +| `ResourceAttribute` | `PRESENT \| INITIALIZED \| TESTED` (0x07) | Must have all three tested-memory flags. | +| `PhysicalStart` | Address | Base address of the bin region. | +| `ResourceLength` | Size in bytes | Must be ≥ total bin size (sum of aligned page counts). | + +**Validation rules:** + +- Exactly one Resource Descriptor HOB with this owner GUID must exist. Multiple are rejected. +- `ResourceLength` must be large enough to hold all bins (checked via `calculate_total_bin_size()`). +- If this HOB is present and valid, bins use the provided range (Path A). If absent, DXE allocates + its own range (Path B). + +#### 3. Memory Allocation HOBs with Name GUID (optional) + +Memory Allocation HOBs (`EFI_HOB_TYPE_MEMORY_ALLOCATION`) whose `Name` field in the allocation +descriptor matches `gEfiMemoryTypeInformationGuid`. These are produced by PEI's bin-aware allocator +to mark runtime allocations that PEI made within the bin region. + +```text ++----------+------------------------------+-----------+-----------+------+---+ +| Header | Name | MemBase | MemLen | Type |...| +| (Hob) | 4C19049F-...-A83FFDFA | | | | | ++----------+------------------------------+-----------+-----------+------+---+ +``` + +**Fields used:** + +| Field | Description | +|---------------------|-----------------------------------------------------------------------------| +| `Name` | `gEfiMemoryTypeInformationGuid` - marks this as a PEI bin-aware allocation. | +| `MemoryBaseAddress` | Physical address of the allocation. | +| `MemoryLength` | Size in bytes (converted to pages). | +| `MemoryType` | `efi::MemoryType` of the allocation. | + +Patina iterates these HOBs after bin initialization and calls `seed_statistics_from_hob()` for each. +If the allocation falls within the type's bin range, `current_number_of_pages()` is incremented to +account for PEI-phase bin usage. Allocations that fall outside all bin ranges are not counted. + +Memory Allocation HOBs without the `gEfiMemoryTypeInformationGuid` name are processed normally by +`process_hob_allocations()` but are not included in bin statistics. + +### Initialization + +Bin initialization runs once during `init_memory_support()`, after the GCD and pre-DXE HOB allocations have been fully +processed. + +The initialization flow resolves a contiguous bin range and then subdivides it into per-type bins. The range is +resolved in priority order: + +1. PEI bins (Path A): Use a pre-allocated range from an incoming Resource Descriptor HOB. +2. DXE bins (Path B): Designate a single contiguous block from the GCD. + +### Path A: PEI provided bin range + +If PEI allocated memory bins (indicated by a Resource Descriptor HOB with an owner GUID of +`gEfiMemoryTypeInformationGuid`), Patina uses that pre-allocated range directly: + +- The Resource Descriptor HOB must describe `EFI_RESOURCE_SYSTEM_MEMORY` with `PRESENT | INITIALIZED | TESTED` + attributes. +- Exactly one such HOB must exist. If multiple are found, all are rejected since this request would be ambiguous. +- The range must be large enough to fit all bins (including alignment padding). +- Bins are divided within the range from the top address downward, with each bin aligned to its type's allocation + granularity (64 KB for runtime types on AArch64, 4 KB otherwise). + +This path provides the most resilience for hibernate stability because the bin region lives at the same physical +address every boot (the platform controls where it is placed in PEI). + +After bin ranges are established, Patina scans Memory Allocation HOBs whose `Name` field matches +`gEfiMemoryTypeInformationGuid`. These are allocations PEI's bin-aware allocator made. For each one that falls within +a bin range, the bin's `current_number_of_pages` is incremented to seed the statistics with pre-DXE usage. + +### Path B: DXE-allocated bins + +If no Resource Descriptor HOB is found, Patina allocates a single contiguous block from the GCD that is large enough +to hold all bins plus worst-case alignment padding: + +1. A conservative total size is calculated which is the sum of all entry sizes plus one unit of `max_granularity` per + entry for alignment padding, rounded up to `max_granularity`. +2. The block is allocated with `GCD.allocate_memory_space()` with alignment matching the maximum granularity + across all bin types (`MemoryBinManager::max_granularity()`). +3. The block is immediately freed back to the GCD so it can be re-claimed. +4. The block is then subdivided into per-type bins using the same logic as Path A. + +This path provides less stability than Path A because the bin addresses depend on the GCD allocator's state at +the time of initialization, which has a relatively greater chance to vary between boots. + +### Integration with Per-Type Allocators + +Patina uses a per-type allocator model where each EFI memory type has its own `FixedSizeBlockAllocator` that manages +a pool of pages obtained from the GCD. Two allocation paths must respect bin boundaries: + +1. UEFI API path: `EFI_BOOT_SERVICES.AllocatePages()` calls `core_allocate_pages()`, which delegates to the + per-type allocator's `allocate_pages()`. +2. Internal expansion path: Pool allocations (`AllocatePool`), Rust heap allocations (`Vec`, `Box`), and + allocator expansion call `GCD.allocate_memory_space()` directly from within the `SpinLockedFixedSizeBlockAllocator`, + bypassing `core_allocate_pages()`. + +Two mechanisms ensure allocations land in bins and bins are protected: + +#### GCD Ownership Protection + +During bin initialization, bin pages are allocated then freed with `free_memory_space_preserving_ownership()`. The +pages become free but retain the per-type allocator's handle as the GCD owner. Other allocators using +`AllocateRespectingOwnership` (the default strategy) skip blocks owned by a different handle, preventing +cross-type intrusion at the GCD level. + +#### Bin-Preference Allocation + +The `reserved_range` set on each per-type `SpinLockedFixedSizeBlockAllocator` enables two behaviors: + +- Allocation preference: `allocate_from_gcd()` first attempts `TopDown(bin_end)` to allocate within the bin + range. If the result lands within the bin, it is used immediately. If the bin is full (the result lands below + the bin base), the allocation is freed and retried with the original strategy. This preference applies to + unconstrained strategies (`TopDown(None)`, `BottomUp(None)`) and constrained strategies + (`TopDown(Some(max))`, `BottomUp(Some(max))`) when the max address is at or above the bin range. `Address` + strategies are never redirected. This ensures that `AllocateMaxAddress` calls from DXE drivers also land in + their designated bins when possible, preventing fragmented special-type allocations outside bin ranges. +- Ownership-preserving free: `free_pages()` checks `in_reserved_range()`. Pages within the bin are freed with + `free_memory_space_preserving_ownership()`, retaining the GCD ownership handle so the bin pages remain protected + after free. + +### Statistics Tracking + +`core_allocate_pages()` and `core_free_pages()` record each operation in the bin manager for special memory types only +(runtime, reserved, ACPI, and PAL types that persist into the OS). Non-special types like `BootServicesCode` and +`BootServicesData` are skipped. Their allocations do not affect bin descriptors or BDS recommendations, so tracking them +would be pure overhead. + +For tracked types: + +- Every allocation and free for the type increments/decrements the "number of pages". +- If `current_number_of_pages` exceeds the previous peak, the memory type information entry is updated. + BDS can use this value to recommend a larger bin size for the next boot. + +Internal allocator expansion (pool growth, Rust heap allocations) is not individually tracked in the bin +statistics. This is consistent with edk2, where pool allocations are only visible through the page-level +expansion events they trigger. + +### GetMemoryMap() Bin Descriptors + +When `GetMemoryMap()` is called, the GCD populates the memory map as usual, then the bin manager post-processes the +buffer. For each "special" memory type with an active bin: + +1. Find `EfiConventionalMemory` entries that overlap the bin range. +2. Convert entries fully within the bin to the bin's memory type. +3. Split entries that partially overlap at the bin boundaries. +4. Set `EFI_MEMORY_RUNTIME` on entries for runtime types. + +This ensures the OS sees a single large descriptor for each bin type, regardless of how much of the bin is actually +allocated. Free space within the bin is reported as the bin type rather than as conventional memory. + +The buffer size calculation in `get_memory_map()` accounts for the worst-case number of additional entries from bin +splitting (2 extra entries per active bin). + +### Config Table + +The bin manager's memory type information is published as the `gMemoryTypeInformationGuid` config table via +`install_memory_type_info_table()`. BDS consumes this table to decide whether bin sizes need adjustment. + +The config table data comes from a fixed-size `[EFiMemoryTypeInformation]` array inside the `MemoryBinManager` +static. It is populated from the HOB during initialization with the original HOB values. `record_allocation()` +updates entries when in-bin usage exceeds the original value, creating a monotonically increasing high-water mark. + +If bins are not initialized (no Memory Type Information HOB was present), the config table is not installed. + +## Comparison with edk2 + +Since both the edk2 DXE Core and Patina DXE Core implement PEI memory bin support, this section makes comparison of +the two implementations easier by summarizing the key design and implementation differences in one place. + +PEI bin support is provided by the PEI Core (C code in edk2), the Patina DXE Core consumes the HOBs that PEI produces. + +| Aspect | edk2 | Patina | +|--------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------| +| Primary Implementation | Shared implementation in `MemoryBin.c` (C, used by both PEI and DXE cores) | Not applicable. Patina is DXE-only. PEI bin logic lives in edk2's PEI Core. | +| Bin state storage | Global arrays `mMemoryTypeStatistics[]` and `gMemoryTypeInformation[]` | `MemoryBinManager` struct behind a `TplMutex` static. Statistics and memory type info are fixed-size arrays inside the struct. | +| Allocation steering | `FindFreePages()` checks `mMemoryTypeStatistics[Type].MaximumAddress` / `BaseAddress` first, falls back to `mDefaultMaximumAddress`, then full range. | `allocate_from_gcd()` tries `TopDown(bin_end)` first for unconstrained and reachable constrained strategies, falls back to the original strategy. | +| GetMemoryMap() bin descriptors | Inline loop in `CoreGetMemoryMap()` splits and converts `EfiConventionalMemory` entries overlapping bins. | `apply_bin_descriptors()` post-processes the buffer after `populate_efi_memory_map()`. Same splitting logic. | +| Statistics update | `UpdateMemoryStatistics()` counts allocations in the bin range or in the "default range". | `record_allocation()` / `record_free()` count all special-type allocations in and outside the bin range. | +| PEI HOB seeding | `InitializeBinStatisticsFromRange()` processes Memory Allocation HOBs with the `gEfiMemoryTypeInformationGuid` name. | `seed_bin_statistics_from_hobs()` performs the same scan and calls `seed_statistics_from_hob()` for each matching HOB. | +| Memory type info storage | `gMemoryTypeInformation[]` is a global C array. | Fixed-size `[EFiMemoryTypeInformation]` array inside the `MemoryBinManager` static, to allow pointers to a stable memory location. | + +### What Patina Does Not Implement + +- PEI bin allocation. PEI bin setup, PCD-based opt-in, PHIT updates, and Memory Allocation HOB marking are all + handled by the pre-DXE environment. +- BDS heuristics. The bin size recommendation logic and `PcdResetOnMemoryTypeInformationChange` reboot are in + BDS code outside of the DXE Core. + +## Logging + +All memory bin log messages use the `memory_bin` log target. These are some key messages and their log levels: + +| Level | Message | When | +|---------|----------------------------------------|--------------------------------------| +| `info` | Memory Type Information HOB found | HOB extraction during init | +| `info` | Bin layout per type (base, max, pages) | Bin initialization | +| `info` | Bins allocated/initialized from range | Initialization complete | +| `debug` | PEI seed per allocation HOBs | Statistics seeding | +| `debug` | GetMemoryMap() bin processing | Each bin processed in GetMemoryMap() | +| `trace` | Individual alloc/free recording | Every page allocation/free | +| `trace` | Bin stats old -> new transitions | Every record_allocation/record_free | +| `trace` | Bin table peak updates | When peak exceeds previous value | + +Filter with the `memory_bin` log target to isolate bin-related output. diff --git a/patina_dxe_core/src/allocator.rs b/patina_dxe_core/src/allocator.rs index 5e24f1e9f..440e03275 100644 --- a/patina_dxe_core/src/allocator.rs +++ b/patina_dxe_core/src/allocator.rs @@ -30,6 +30,7 @@ use crate::{ GCD, config_tables, gcd::{self, AllocateType as AllocationStrategy}, memory_attributes_table::MemoryAttributesTable, + memory_bin::MemoryBinManager, protocol_db::{self, INVALID_HANDLE}, protocols::PROTOCOL_DB, systemtables::EfiSystemTable, @@ -45,6 +46,7 @@ pub use uefi_allocator::UefiAllocator; use patina::{ base::{SIZE_4KB, UEFI_PAGE_MASK, UEFI_PAGE_SIZE}, + efi_types::EFI_MAX_MEMORY_TYPE, error::EfiError, guids, uefi_size_to_pages, }; @@ -86,6 +88,9 @@ const _: () = assert!( const PRIVATE_ALLOCATOR_TRACKING_GUID: patina::BinaryGuid = patina::BinaryGuid::from_string("9D1FA6E9-0C86-4F7F-A99B-DD229C9B3893"); +static MEMORY_BIN_MANAGER: tpl_mutex::TplMutex = + tpl_mutex::TplMutex::new(TPL_HIGH_LEVEL, MemoryBinManager::new(), "MemoryBinManagerLock"); + pub(crate) const DEFAULT_PAGE_ALLOCATION_GRANULARITY: usize = SIZE_4KB; // Per the UEFI spec, AARCH64 runtime pages need to be allocated on 64KB boundaries in units of 64KB to accommodate @@ -153,8 +158,8 @@ pub trait PageAllocator { /// Caller must ensure the address corresponds to a valid block allocated with [`Self::allocate_pages`]. unsafe fn free_pages(&self, address: usize, pages: usize) -> Result<(), EfiError>; - /// Reserves a range of memory for this allocator. - fn reserve_memory_pages(&self, pages: usize) -> Result<(), EfiError>; + /// Sets the reserved memory range (bin range) for this allocator. + fn set_reserved_range(&self, range: Range); /// Returns an iterator over the memory ranges managed by this allocator. fn get_memory_ranges(&self) -> alloc::vec::IntoIter>; @@ -185,7 +190,7 @@ pub(crate) static EFI_BOOT_SERVICES_DATA_ALLOCATOR: UefiAllocatorWithFsb = UefiA SpinLockedFixedSizeBlockAllocator::new( &GCD, protocol_db::EFI_BOOT_SERVICES_DATA_ALLOCATOR_HANDLE, - NonNull::from_ref(GCD.memory_type_info(efi::BOOT_SERVICES_DATA)), + efi::BOOT_SERVICES_DATA, DEFAULT_PAGE_ALLOCATION_GRANULARITY, HIGH_TRAFFIC_ALLOC_MIN_EXPANSION, ), @@ -199,7 +204,7 @@ pub static EFI_LOADER_CODE_ALLOCATOR: UefiAllocatorWithFsb = UefiAllocator::new( SpinLockedFixedSizeBlockAllocator::new( &GCD, protocol_db::EFI_LOADER_CODE_ALLOCATOR_HANDLE, - NonNull::from_ref(GCD.memory_type_info(efi::LOADER_CODE)), + efi::LOADER_CODE, DEFAULT_PAGE_ALLOCATION_GRANULARITY, HIGH_TRAFFIC_ALLOC_MIN_EXPANSION, ), @@ -210,7 +215,7 @@ pub static EFI_LOADER_DATA_ALLOCATOR: UefiAllocatorWithFsb = UefiAllocator::new( SpinLockedFixedSizeBlockAllocator::new( &GCD, protocol_db::EFI_LOADER_DATA_ALLOCATOR_HANDLE, - NonNull::from_ref(GCD.memory_type_info(efi::LOADER_DATA)), + efi::LOADER_DATA, DEFAULT_PAGE_ALLOCATION_GRANULARITY, HIGH_TRAFFIC_ALLOC_MIN_EXPANSION, ), @@ -221,33 +226,29 @@ pub static EFI_BOOT_SERVICES_CODE_ALLOCATOR: UefiAllocatorWithFsb = UefiAllocato SpinLockedFixedSizeBlockAllocator::new( &GCD, protocol_db::EFI_BOOT_SERVICES_CODE_ALLOCATOR_HANDLE, - NonNull::from_ref(GCD.memory_type_info(efi::BOOT_SERVICES_CODE)), + efi::BOOT_SERVICES_CODE, DEFAULT_PAGE_ALLOCATION_GRANULARITY, LOW_TRAFFIC_ALLOC_MIN_EXPANSION, ), efi::BOOT_SERVICES_CODE, ); -// This needs to call MemoryAttributesTable::install on allocation/deallocation, hence having the real callback -// passed in pub static EFI_RUNTIME_SERVICES_CODE_ALLOCATOR: UefiAllocatorWithFsb = UefiAllocator::new( SpinLockedFixedSizeBlockAllocator::new( &GCD, protocol_db::EFI_RUNTIME_SERVICES_CODE_ALLOCATOR_HANDLE, - NonNull::from_ref(GCD.memory_type_info(efi::RUNTIME_SERVICES_CODE)), + efi::RUNTIME_SERVICES_CODE, RUNTIME_PAGE_ALLOCATION_GRANULARITY, LOW_TRAFFIC_RUNTIME_ALLOC_MIN_EXPANSION, ), efi::RUNTIME_SERVICES_CODE, ); -// This needs to call MemoryAttributesTable::install on allocation/deallocation, hence having the real callback -// passed in pub static EFI_RUNTIME_SERVICES_DATA_ALLOCATOR: UefiAllocatorWithFsb = UefiAllocator::new( SpinLockedFixedSizeBlockAllocator::new( &GCD, protocol_db::EFI_RUNTIME_SERVICES_DATA_ALLOCATOR_HANDLE, - NonNull::from_ref(GCD.memory_type_info(efi::RUNTIME_SERVICES_DATA)), + efi::RUNTIME_SERVICES_DATA, RUNTIME_PAGE_ALLOCATION_GRANULARITY, LOW_TRAFFIC_RUNTIME_ALLOC_MIN_EXPANSION, ), @@ -485,20 +486,11 @@ impl AllocatorMap { _ => UEFI_PAGE_SIZE, }; - // If this is one of the memory types tracked by the system table, we will use the memory type info struct - // from the GCD. Otherwise, we will just leak a new memory type info struct with the given memory type and - // have the allocator use it. - let memory_type_info = if (memory_type as usize) <= GCD.memory_type_info_table().len() { - NonNull::from_ref(GCD.memory_type_info(memory_type)) - } else { - NonNull::from_ref(Box::leak(Box::new(EFiMemoryTypeInformation { memory_type, number_of_pages: 0 }))) - }; - Box::leak(Box::new(UefiAllocator::new( SpinLockedFixedSizeBlockAllocator::new( &GCD, handle, - memory_type_info, + memory_type, granularity, LOW_TRAFFIC_RUNTIME_ALLOC_MIN_EXPANSION, ), @@ -718,6 +710,11 @@ pub fn core_allocate_pages( _ => {} } + // Record the allocation in the memory bin manager for bin statistics tracking. + if res.is_ok() { + MEMORY_BIN_MANAGER.lock().record_allocation(memory_type, pages as u64); + } + res } @@ -787,6 +784,11 @@ pub fn core_free_pages(memory: efi::PhysicalAddress, pages: usize) -> Result<(), _ => {} } + // Record the free in the memory bin manager for bin statistics tracking. + if res.is_ok() { + MEMORY_BIN_MANAGER.lock().record_free(memory_type, pages as u64); + } + res } @@ -827,7 +829,11 @@ extern "efiapi" fn get_memory_map( // SAFETY: caller must ensure that memory_map_size is a valid pointer. It is null-checked above. let map_size = unsafe { memory_map_size.read_unaligned() }; - let required_map_size = GCD.memory_descriptor_count_for_efi_memory_map() * mem::size_of::(); + // Account for additional descriptors that may be produced during bin splitting. + let bin_extra_descriptors = MEMORY_BIN_MANAGER.lock().max_additional_descriptors(); + let base_descriptor_count = GCD.memory_descriptor_count_for_efi_memory_map(); + let total_descriptor_count = base_descriptor_count + bin_extra_descriptors; + let required_map_size = total_descriptor_count * mem::size_of::(); debug_assert!(required_map_size != 0); if required_map_size == 0 { return efi::Status::NOT_FOUND; @@ -852,6 +858,10 @@ extern "efiapi" fn get_memory_map( Ok(count) => count, Err(err) => return err.into(), }; + + // Apply memory bin descriptors: convert free memory within bin regions to the bin's type. + let actual_count = MEMORY_BIN_MANAGER.lock().apply_bin_descriptors(buffer, actual_count); + let actual_map_size = actual_count * mem::size_of::(); // Write back the actual map size after merging @@ -872,9 +882,55 @@ extern "efiapi" fn get_memory_map( log::debug!(target: "efi_memory_map", "EFI_MEMORY_MAP: \n{:?}", MemoryDescriptorSlice(&buffer[..actual_count])); + if log::log_enabled!(target: "memory_bin", log::Level::Debug) { + dump_memory_bin_stats(); + } + + if log::log_enabled!(target: "allocations", log::Level::Trace) { + dump_allocator_details(); + } + efi::Status::SUCCESS } +/// Dumps bin manager peak tracking data at debug level. +#[coverage(off)] +fn dump_memory_bin_stats() { + let bin_manager = MEMORY_BIN_MANAGER.lock(); + if bin_manager.is_initialized() { + log::debug!(target: "memory_bin", "Bin manager table (peak tracking for BDS):"); + for entry in bin_manager.memory_type_information() { + log::debug!( + target: "memory_bin", + " {} pages={}", + crate::memory_bin::memory_type_name(entry.memory_type), + entry.number_of_pages + ); + } + } +} + +/// Dumps per-allocator page counts at trace level. +#[coverage(off)] +fn dump_allocator_details() { + log::trace!(target: "allocations", "Allocator page counts:"); + for (alloc, _) in STATIC_ALLOCATORS.iter() { + let stats = alloc.stats(); + let reserved_free = uefi_size_to_pages!(stats.reserved_size - stats.reserved_used); + let net_pages = stats.claimed_pages.saturating_sub(reserved_free); + if net_pages > 0 { + log::trace!( + target: "allocations", + " {} net_pages={} (claimed={} reserved_free={})", + crate::memory_bin::memory_type_name(alloc.memory_type()), + net_pages, + stats.claimed_pages, + reserved_free + ); + } + } +} + pub fn terminate_memory_map(map_key: usize) -> Result<(), EfiError> { match GCD.get_last_efi_memory_map_key() { Some(key) if key == map_key => Ok(()), @@ -883,7 +939,14 @@ pub fn terminate_memory_map(map_key: usize) -> Result<(), EfiError> { } pub fn install_memory_type_info_table(system_table: &mut EfiSystemTable) -> Result<(), EfiError> { - let table_ptr = NonNull::from(GCD.memory_type_info_table()).cast::().as_ptr(); + let bin_manager = MEMORY_BIN_MANAGER.lock(); + if !bin_manager.is_initialized() || bin_manager.memory_type_information().is_empty() { + log::warn!(target: "memory_bin", "No bin manager data available. Memory type information config table not installed."); + return Ok(()); + } + let table_ptr = bin_manager.memory_type_information().as_ptr() as *mut c_void; + drop(bin_manager); + config_tables::core_install_configuration_table( guids::MEMORY_TYPE_INFORMATION.into_inner(), table_ptr, @@ -1121,59 +1184,315 @@ pub fn init_memory_support(hob_list: &HobList) { // reserved. gcd::add_hob_resource_descriptors_to_gcd(hob_list); - // process pre-DXE allocations from the Hob list + // Process pre-DXE allocations from the Hob list process_hob_allocations(hob_list); // After this point the GCD and existing allocations are fully processed and it is safe to arbitrarily allocate. - // If memory type info HOB is available, then pre-allocate the corresponding buckets. - if let Some(memory_type_info) = hob_list.iter().find_map(|x| { - match x { - patina::pi::hob::Hob::GuidHob(hob, data) if hob.name == MEMORY_TYPE_INFO_HOB_GUID.into_inner() => { - let memory_type_slice_ptr = data.as_ptr() as *const EFiMemoryTypeInformation; - let memory_type_slice_len = data.len() / mem::size_of::(); + // Initialize memory bins from the Memory Type Information HOB if present. + if let Some(memory_type_info) = crate::memory_bin::extract_memory_type_info_from_hob(hob_list) { + log::info!(target: "memory_bin", "Memory Type Information HOB found with {} entries.", memory_type_info.len()); - // SAFETY: this structure comes from the hob list, so it must be 8-byte aligned per the PI spec. - // A compile-time assertion above guarantees EFiMemoryTypeInformation's alignment requirement - // is <= 8 bytes, so alignment is always satisfied. Length is calculated above to fit within - // the Guid HOB data. - let memory_type_info = unsafe { slice::from_raw_parts(memory_type_slice_ptr, memory_type_slice_len) }; + initialize_memory_bins(hob_list, &memory_type_info); + seed_bin_statistics_from_hobs(hob_list); + } +} - Some(memory_type_info) - } - _ => None, +/// Initializes memory bins from HOB data. +/// +/// Bins are initialized from a contiguous address range and then claimed per-type via +/// `reserve_bin_ranges`. The range is resolved in priority order: +/// +/// 1. PEI-provided bins: A Resource Descriptor HOB with the `MemoryTypeInformation` owner GUID +/// provides a pre-allocated range from PEI. +/// 2. DXE-allocated bins: If no PEI range exists, a single contiguous block is allocated from the +/// GCD, freed back, and used as the range. +/// +/// Note: A local `MemoryBinManager` is used during initialization to avoid holding the global lock +/// during GCD allocations (which would cause re-entrant lock panics since allocation recording also +/// acquires the lock). +#[coverage(off)] +fn initialize_memory_bins(hob_list: &HobList, memory_type_info: &[EFiMemoryTypeInformation]) { + if MEMORY_BIN_MANAGER.lock().is_initialized() { + return; + } + + // Resolve the overall bin block to either PEI-provided or DXE-allocated. + let bin_range = + find_pei_bin_range(hob_list, memory_type_info).or_else(|| allocate_contiguous_bin_range(memory_type_info)); + + let Some((start, length)) = bin_range else { + log::warn!(target: "memory_bin", "No bin range available. Memory bins will not be initialized."); + return; + }; + + let mut local_manager = MemoryBinManager::new(); + if !local_manager.initialize_from_range(start, length, memory_type_info) { + log::warn!(target: "memory_bin", "Failed to initialize bins from range at {:#X}, length {:#X}.", start, length); + return; + } + + *MEMORY_BIN_MANAGER.lock() = local_manager; + reserve_bin_ranges(); +} + +/// Attempts to find a PEI-provided bin range from a Resource Descriptor HOB. +#[coverage(off)] +fn find_pei_bin_range( + hob_list: &HobList, + memory_type_info: &[EFiMemoryTypeInformation], +) -> Option<(efi::PhysicalAddress, u64)> { + let (start, length) = crate::memory_bin::find_memory_type_info_resource_hob(hob_list, memory_type_info)?; + log::info!(target: "memory_bin", "Found PEI bin region at {:#X}, length {:#X}.", start, length); + Some((start, length)) +} + +/// Allocates a single contiguous block from the GCD for all bin types. +/// +/// The block is freed back to the GCD immediately so that it can be reclaimed for per-type ranges. +#[coverage(off)] +fn allocate_contiguous_bin_range(memory_type_info: &[EFiMemoryTypeInformation]) -> Option<(efi::PhysicalAddress, u64)> { + log::info!(target: "memory_bin", "No PEI bin region found. Allocating a contiguous bin range from the GCD."); + + let alloc_size = MemoryBinManager::contiguous_alloc_size(memory_type_info)?; + let max_granularity = MemoryBinManager::max_granularity(memory_type_info); + let align_shift = max_granularity.trailing_zeros() as usize; + + let temp_handle = memory_type_info + .iter() + .find(|e| e.number_of_pages > 0 && (e.memory_type as usize) < EFI_MAX_MEMORY_TYPE) + .and_then(|e| AllocatorMap::handle_for_memory_type(e.memory_type).ok())?; + + let addr = GCD + .allocate_memory_space( + DEFAULT_ALLOCATION_STRATEGY, + GcdMemoryType::SystemMemory, + align_shift, + alloc_size, + temp_handle, + None, + ) + .inspect_err(|err| { + log::warn!( + target: "memory_bin", + "Failed to allocate contiguous bin range ({:#X} bytes) from GCD: {:?}", + alloc_size, + err + ); + }) + .ok()?; + + // Free the block so reserve_bin_ranges can re-claim per-type ownership. + let _ = GCD.free_memory_space(addr, alloc_size); + + Some((addr as efi::PhysicalAddress, alloc_size as u64)) +} + +/// Seeds bin statistics from PEI Memory Allocation HOBs marked with `MEMORY_TYPE_INFO_HOB_GUID`. +/// +/// These HOBs indicate allocations made by PEI's bin-aware allocator and need to be accounted +/// for in the bin statistics. +fn seed_bin_statistics_from_hobs(hob_list: &HobList) { + let mut bin_manager = MEMORY_BIN_MANAGER.lock(); + let mut seeded_count = 0u32; + + for hob_entry in hob_list.iter() { + let desc = match hob_entry { + Hob::MemoryAllocation(hob::MemoryAllocation { header: _, alloc_descriptor: desc }) + | Hob::MemoryAllocationModule(hob::MemoryAllocationModule { + header: _, + alloc_descriptor: desc, + module_name: _, + entry_point: _, + }) if desc.name == MEMORY_TYPE_INFO_HOB_GUID.into_inner() => desc, + _ => continue, + }; + + if desc.memory_type == efi::CONVENTIONAL_MEMORY || desc.memory_length == 0 { + continue; } - }) { - for bucket in memory_type_info { - if bucket.number_of_pages == 0 { + + let pages = uefi_size_to_pages!(desc.memory_length as usize) as u64; + bin_manager.seed_statistics_from_hob(desc.memory_type, pages); + seeded_count += 1; + } + + log::info!(target: "memory_bin", "Seeded bin statistics from {} PEI Memory Allocation HOBs.", seeded_count); +} + +/// Claims free GCD pages within each bin range for the corresponding bin type's allocator. +/// +/// For each active bin, walks the GCD blocks within the bin's address range and: +/// - Free blocks are claimed with `allocate_memory_space()` at the exact address with the bin +/// type's allocator handle, then `free_memory_space_preserving_ownership()` is used to release the +/// pages while retaining GCD ownership. This prevents other allocators from expanding into the bin range. +/// - Same-type allocated blocks are logged as expected (PEI allocations within the bin). +/// - Different-type allocated blocks are logged as an error with `debug_assert` (should not happen, but +/// the allocation is left in place). +/// - Partial overlaps at bin boundaries are logged as an error with `debug_assert`. +fn reserve_bin_ranges() { + let bin_manager = MEMORY_BIN_MANAGER.lock(); + let bins: Vec<(efi::MemoryType, efi::PhysicalAddress, efi::PhysicalAddress, u64)> = + bin_manager.active_bins().collect(); + drop(bin_manager); + + for (memory_type, bin_base, bin_max, bin_pages) in bins { + let bin_end = bin_max + 1; // exclusive end + let bin_len = (bin_end - bin_base) as usize; + let mut claimed_pages: usize = 0; + let mut same_type_pages: usize = 0; + let mut conflicting_pages: usize = 0; + + let handle = match AllocatorMap::handle_for_memory_type(memory_type) { + Ok(h) => h, + Err(err) => { + log::error!( + target: "memory_bin", + "Failed to get handle for bin[{}] {}: {:?}", + memory_type, + crate::memory_bin::memory_type_name(memory_type), + err + ); continue; } - log::info!( - "Allocating memory bucket for memory type: {:#x?}, {:#x?} pages.", - bucket.memory_type, - bucket.number_of_pages - ); - let handle = match AllocatorMap::handle_for_memory_type(bucket.memory_type) { - Ok(handle) => handle, - Err(err) => { - log::error!("failed to get a handle for memory type {:#x?}: {:#x?}", bucket.memory_type, err); + }; + + log::info!( + target: "memory_bin", + "Reserving bin[{}] {} range=[{:#X}..{:#X}] ({} pages)", + memory_type, + crate::memory_bin::memory_type_name(memory_type), + bin_base, + bin_max, + bin_pages + ); + + for desc_result in GCD.iter(bin_base as usize, bin_len) { + let desc = match desc_result { + Ok(d) => d, + Err(_) => continue, + }; + + // Compute the overlap between this GCD block and the bin range. + let block_start = desc.base_address.max(bin_base); + let block_end = (desc.base_address + desc.length).min(bin_end); + if block_start >= block_end { + continue; + } + let block_len = (block_end - block_start) as usize; + let block_pages = uefi_size_to_pages!(block_len); + + // Log partial overlaps with allocated blocks of a different type. + if (desc.base_address < bin_base || (desc.base_address + desc.length) > bin_end) + && desc.image_handle != INVALID_HANDLE + { + let allocated_type = memory_type_for_handle(desc.image_handle); + if let Some(alloc_type) = allocated_type + && alloc_type != memory_type + { + let desc_end = desc.base_address + desc.length; + log::error!( + target: "memory_bin", + "Partial overlap in bin[{}] {}: GCD block [{:#X}..{:#X}) extends beyond bin [{:#X}..{:#X}), type={}", + memory_type, + crate::memory_bin::memory_type_name(memory_type), + desc.base_address, + desc_end, + bin_base, + bin_end, + crate::memory_bin::memory_type_name(alloc_type), + ); + debug_assert!( + false, + "Partial overlap in bin[{}]: GCD block [{:#X}..{:#X}) extends beyond bin [{:#X}..{:#X})", + memory_type, desc.base_address, desc_end, bin_base, bin_end, + ); + conflicting_pages += block_pages; continue; } - }; + } - match ALLOCATORS.lock().get_or_create_allocator(bucket.memory_type, handle) { - Ok(allocator) => { - if let Err(err) = allocator.reserve_memory_pages(bucket.number_of_pages as usize) { - log::error!("failed to reserve pages for memory type {:#x?}: {:#x?}", bucket.memory_type, err); - continue; + if desc.image_handle == INVALID_HANDLE && desc.memory_type == GcdMemoryType::SystemMemory { + // Free block within the bin range by claiming it at the exact address via GCD with + // ownership preservation. This allocates the pages with the bin type's allocator + // handle, then frees them while retaining the handle so other allocators cannot + // expand into these pages. + match GCD.allocate_memory_space( + AllocationStrategy::Address(block_start as usize), + GcdMemoryType::SystemMemory, + 0, + block_len, + handle, + None, + ) { + Ok(_) => { + if let Err(err) = GCD.free_memory_space_preserving_ownership(block_start as usize, block_len) { + log::error!( + target: "memory_bin", + "Failed to free-with-ownership bin pages at {:#X}: {:?}", + block_start, + err + ); + } + claimed_pages += block_pages; + } + Err(err) => { + log::warn!( + target: "memory_bin", + "Failed to claim bin pages at {:#X} ({} pages): {:?}", + block_start, + block_pages, + err + ); } } - Err(err) => { - log::error!("failed to get an allocator for memory type {:#x?}: {:#x?}", bucket.memory_type, err); - continue; + } else if desc.image_handle != INVALID_HANDLE { + // Check if the allocated block is the correct type. + let allocated_type = memory_type_for_handle(desc.image_handle); + if let Some(alloc_type) = allocated_type { + if alloc_type == memory_type { + same_type_pages += block_pages; + } else { + conflicting_pages += block_pages; + log::error!( + target: "memory_bin", + "Conflicting allocation in bin[{}] {} range: [{:#X}..{:#X}) is {} (expected {})", + memory_type, + crate::memory_bin::memory_type_name(memory_type), + block_start, + block_end, + crate::memory_bin::memory_type_name(alloc_type), + crate::memory_bin::memory_type_name(memory_type), + ); + debug_assert!( + false, + "Conflicting allocation in bin[{}] range: [{:#X}..{:#X}) is {} (expected {})", + memory_type, + block_start, + block_end, + crate::memory_bin::memory_type_name(alloc_type), + crate::memory_bin::memory_type_name(memory_type), + ); + } } } } + + log::info!( + target: "memory_bin", + "Reserved bin[{}] {}: claimed={} pages, existing={} pages, conflicting={} pages (of {} total)", + memory_type, + crate::memory_bin::memory_type_name(memory_type), + claimed_pages, + same_type_pages, + conflicting_pages, + bin_pages + ); + + // Set the reserved range so allocations prefer the bin and frees preserve ownership. + if let Ok(allocator) = ALLOCATORS.lock().get_or_create_allocator(memory_type, handle) { + allocator.set_reserved_range(bin_base..bin_end); + } } } @@ -1198,6 +1517,7 @@ pub(crate) unsafe fn reset_allocators() { // allocations are active. A lock is used to ensure exclusive access preventing // use while reset occurs. unsafe { ALLOCATORS.lock().reset() }; + MEMORY_BIN_MANAGER.lock().reset(); } #[cfg(test)] @@ -1270,6 +1590,20 @@ mod tests { .unwrap(); } + /// Finds a bin by memory type from a collected `active_bins()` list and returns `(base, max)`. + /// + /// Panics with a descriptive message if the bin is not found. + fn find_bin_range( + bins: &[(efi::MemoryType, efi::PhysicalAddress, efi::PhysicalAddress, u64)], + memory_type: efi::MemoryType, + ) -> (efi::PhysicalAddress, efi::PhysicalAddress) { + let (_, base, max, _) = bins + .iter() + .find(|(mt, _, _, _)| *mt == memory_type) + .unwrap_or_else(|| panic!("Expected bin for memory type {:#X} not found", memory_type)); + (*base, *max) + } + #[test] #[allow(unpredictable_function_pointer_comparisons)] fn install_memory_support_should_populate_boot_services_ptrs() { @@ -1327,15 +1661,20 @@ mod tests { init_memory_support(&hob_list); - let pal_code_range = ALLOCATORS.lock().get_allocator(efi::PAL_CODE).unwrap().reserved_range().unwrap(); - assert_eq!(pal_code_range.end - pal_code_range.start, 0x100 * 0x1000); + let bin_manager = MEMORY_BIN_MANAGER.lock(); + assert!(bin_manager.is_initialized(), "Bin manager should be initialized"); - let reclaim_range = - ALLOCATORS.lock().get_allocator(efi::ACPI_RECLAIM_MEMORY).unwrap().reserved_range().unwrap(); - assert_eq!(reclaim_range.end - reclaim_range.start, 0x200 * 0x1000); + // Verify bin manager has the expected bin ranges. + let bins: Vec<_> = bin_manager.active_bins().collect(); - let nvs_range = ALLOCATORS.lock().get_allocator(efi::ACPI_MEMORY_NVS).unwrap().reserved_range().unwrap(); - assert_eq!(nvs_range.end - nvs_range.start, 0x300 * 0x1000); + let (pal_base, pal_max) = find_bin_range(&bins, efi::PAL_CODE); + assert_eq!((pal_max - pal_base + 1), 0x100 * 0x1000, "PAL_CODE bin size mismatch"); + + let (reclaim_base, reclaim_max) = find_bin_range(&bins, efi::ACPI_RECLAIM_MEMORY); + assert_eq!((reclaim_max - reclaim_base + 1), 0x200 * 0x1000, "ACPI_RECLAIM_MEMORY bin size mismatch"); + + let (nvs_base, nvs_max) = find_bin_range(&bins, efi::ACPI_MEMORY_NVS); + assert_eq!((nvs_max - nvs_base + 1), 0x300 * 0x1000, "ACPI_MEMORY_NVS bin size mismatch"); }) } @@ -1844,9 +2183,6 @@ mod tests { #[test] fn get_memory_map_should_return_a_memory_map() { with_locked_state(GcdInit::WithSize(0x1000000), |_physical_hob_list| { - //reserve some pages in the runtime services data allocator. - ALLOCATORS.lock().get_allocator(efi::RUNTIME_SERVICES_DATA).unwrap().reserve_memory_pages(0x100).unwrap(); - // allocate some "custom" type pages to create something interesting to find in the map. let mut buffer_ptr: *mut u8 = core::ptr::null_mut(); assert_eq!( diff --git a/patina_dxe_core/src/allocator/fixed_size_block_allocator.rs b/patina_dxe_core/src/allocator/fixed_size_block_allocator.rs index 9478e79eb..8dfce8d70 100644 --- a/patina_dxe_core/src/allocator/fixed_size_block_allocator.rs +++ b/patina_dxe_core/src/allocator/fixed_size_block_allocator.rs @@ -28,9 +28,9 @@ use core::{ }; use linked_list_allocator::{align_down_size, align_up_size}; use patina::{ - base::{UEFI_PAGE_SHIFT, UEFI_PAGE_SIZE, align_up}, + base::{UEFI_PAGE_SIZE, align_up, page_shift_from_alignment}, error::EfiError, - pi::{dxe_services::GcdMemoryType, hob::EFiMemoryTypeInformation}, + pi::dxe_services::GcdMemoryType, uefi_pages_to_size, uefi_size_to_pages, }; use r_efi::efi; @@ -60,16 +60,6 @@ fn list_index(layout: &Layout) -> Option { BLOCK_SIZES.iter().position(|&s| s >= required_block_size) } -/// Converts the given alignment to a shift value. -const fn page_shift_from_alignment(alignment: usize) -> Result { - let shift = alignment.trailing_zeros() as usize; - if !alignment.is_power_of_two() || shift < UEFI_PAGE_SHIFT { - return Err(EfiError::InvalidParameter); - } - - Ok(shift) -} - struct BlockListNode { next: Option<&'static mut BlockListNode>, } @@ -109,9 +99,8 @@ impl Iterator for AllocatorIterator { /// the allocator where a new backing linked-list is created. /// pub struct FixedSizeBlockAllocator { - /// The memory type this allocator is managing and number of pages allocated for this memory type. This is used - /// to bucketize memory for the EFI_MEMORY_MAP and handle any special cases for memory types. - memory_type_info: NonNull, + /// The memory type this allocator manages. + memory_type: efi::MemoryType, /// The heads of the linked lists for each fixed-size block. Each index corresponds to a block size in /// `BLOCK_SIZES`. @@ -136,10 +125,10 @@ pub struct FixedSizeBlockAllocator { impl FixedSizeBlockAllocator { /// Creates a new empty FixedSizeBlockAllocator - pub const fn new(memory_type_info: NonNull, page_allocation_granularity: usize) -> Self { + pub const fn new(memory_type: efi::MemoryType, page_allocation_granularity: usize) -> Self { const EMPTY: Option<&'static mut BlockListNode> = None; FixedSizeBlockAllocator { - memory_type_info, + memory_type, list_heads: [EMPTY; BLOCK_SIZES.len()], allocators: None, reserved_range: None, @@ -156,7 +145,6 @@ impl FixedSizeBlockAllocator { self.list_heads = [EMPTY; BLOCK_SIZES.len()]; self.allocators = None; self.reserved_range = None; - self.memory_type_info_mut().number_of_pages = 0; self.stats = AllocationStatistics::new(); } @@ -207,9 +195,6 @@ impl FixedSizeBlockAllocator { self.stats.claimed_pages += uefi_size_to_pages!(new_region.len()); } - // if we managed to allocate pages, call into the page change callback to update stats - self.update_memory_type_info(); - Ok(()) } @@ -325,31 +310,21 @@ impl FixedSizeBlockAllocator { } } - /// Informs the allocator of it's reserved memory range. - /// - /// This function is intended to be called on a region of memory that has been marked with a backing memory allocator - /// as reserved for this allocator. Calling this funcion does not itself reserve the region of memory. + /// Sets the reserved memory range (bin range) for this allocator. /// - /// ## Safety + /// ## Errors /// - /// The range must not overlap with any existing allocations. - pub fn set_reserved_range(&mut self, range: NonNull<[u8]>) -> Result<(), EfiError> { + /// Returns [`EfiError::AlreadyStarted`] if a reserved range has already been set. + pub fn set_reserved_range(&mut self, range: Range) -> Result<(), EfiError> { if self.reserved_range.is_some() { Err(EfiError::AlreadyStarted)?; } - self.reserved_range = Some( - range.addr().get() as efi::PhysicalAddress - ..range.addr().get() as efi::PhysicalAddress + range.len() as efi::PhysicalAddress, - ); - - self.stats.reserved_size = range.len(); + let size = (range.end - range.start) as usize; + self.reserved_range = Some(range); + self.stats.reserved_size = size; self.stats.reserved_used = 0; - self.stats.claimed_pages += uefi_size_to_pages!(range.len()); - - // call into the page change callback to keep track of the updated reserved stats and - // any memory map changes made when reserving the range. - self.update_memory_type_info(); + self.stats.claimed_pages += uefi_size_to_pages!(size); Ok(()) } @@ -374,9 +349,6 @@ impl FixedSizeBlockAllocator { } else { self.stats.claimed_pages += uefi_size_to_pages!(allocation.len()); } - - // if we managed to allocate pages, call into the page change callback to update stats - self.update_memory_type_info(); } /// Tracks page freeing for record keeping @@ -386,9 +358,6 @@ impl FixedSizeBlockAllocator { } else { self.stats.claimed_pages = self.stats.claimed_pages.saturating_sub(pages); } - - // call into the page change callback to update stats - self.update_memory_type_info(); } /// Get the ranges of the memory owned by this allocator @@ -403,36 +372,16 @@ impl FixedSizeBlockAllocator { }) } - #[inline(always)] - fn memory_type_info(&self) -> &EFiMemoryTypeInformation { - // SAFETY: memory_type_info is a pointer to a leaked MemoryTypeInfo structure and there have been no type casts - unsafe { self.memory_type_info.as_ref() } - } - - #[inline(always)] - fn memory_type_info_mut(&mut self) -> &mut EFiMemoryTypeInformation { - // SAFETY: memory_type_info is a pointer to a leaked MemoryTypeInfo structure and there have been no type casts - unsafe { self.memory_type_info.as_mut() } - } - /// Returns the memory type for this allocator #[inline(always)] pub fn memory_type(&self) -> efi::MemoryType { - self.memory_type_info().memory_type + self.memory_type } /// Returns a reference to the allocation stats for this allocator. pub fn stats(&self) -> &AllocationStatistics { &self.stats } - - /// Re-calculates the number of pages allocated for this memory type and updates the memory type info. - fn update_memory_type_info(&mut self) { - let stats = self.stats(); - let reserved_free = uefi_size_to_pages!(stats.reserved_size - stats.reserved_used); - let page_count = (stats.claimed_pages - reserved_free) as u32; - self.memory_type_info_mut().number_of_pages = page_count; - } } impl Display for FixedSizeBlockAllocator { @@ -493,7 +442,7 @@ impl SpinLockedFixedSizeBlockAllocator { pub const fn new( gcd: &'static SpinLockedGcd, allocator_handle: efi::Handle, - memory_type_info: NonNull, + memory_type: efi::MemoryType, page_allocation_granularity: usize, min_expansion: usize, ) -> Self { @@ -502,7 +451,7 @@ impl SpinLockedFixedSizeBlockAllocator { handle: allocator_handle, inner: tpl_mutex::TplMutex::new( efi::TPL_HIGH_LEVEL, - FixedSizeBlockAllocator::new(memory_type_info, page_allocation_granularity), + FixedSizeBlockAllocator::new(memory_type, page_allocation_granularity), "FsbLock", ), min_expansion, @@ -571,15 +520,7 @@ impl SpinLockedFixedSizeBlockAllocator { // Page allocations and pool allocations are disjoint; page allocations are allocated directly from the GCD and are // freed straight back to GCD. As such, a tracking allocator structure is not required. let start_address = self - .gcd - .allocate_memory_space( - allocation_strategy, - GcdMemoryType::SystemMemory, - align_shift, - uefi_pages_to_size!(required_pages), - self.handle, - None, - ) + .allocate_from_gcd(allocation_strategy, align_shift, uefi_pages_to_size!(required_pages)) .map_err(|err| match err { EfiError::InvalidParameter | EfiError::NotFound => err, _ => EfiError::OutOfResources, @@ -643,49 +584,64 @@ impl SpinLockedFixedSizeBlockAllocator { Ok(()) } - /// Reserves a range of memory to be used by this allocator of the given size in pages. + /// Sets the reserved memory range (bin range) for this allocator. /// - /// The caller specifies a maximum number of pages this allocator is expected to require, and as long as the number - /// of pages actually used by the allocator is less than that amount, then all the allocations for this allocator - /// will be in a single contiguous block. This capability can be used to ensure that the memory map presented to the - /// OS is stable from boot-to-boot despite small boot-to-boot variations in actual page usage. + /// See [`FixedSizeBlockAllocator::set_reserved_range()`] for details on the accounting model. + pub fn set_reserved_range(&self, range: Range) { + let _ = self.lock().set_reserved_range(range); + } + + /// Attempts to allocate from the GCD, preferring the reserved (bin) range if one exists. /// - /// For best memory stability, this routine should be called only during the initialization of the memory subsystem; - /// calling it after other allocations/frees have occurred will not cause allocation errors, but may cause the - /// memory map to vary from boot-to-boot. + /// For strategies that do not exclude the bin range, this first tries to allocate within the + /// bin range so that special-type pages land in their designated bin. Specifically, bin + /// preference is attempted for: + /// - `TopDown(None)` / `BottomUp(None)`: fully unconstrained strategies. + /// - `TopDown(Some(max))` / `BottomUp(Some(max))`: constrained strategies where `max` is at + /// or above the bin range end, meaning the bin is reachable. /// - /// This routine will return Err(efi::Status::ALREADY_STARTED) if it is called more than once. + /// `Address(addr)` strategies are never redirected because the caller requires an exact address. /// - pub fn reserve_memory_pages(&self, pages: usize) -> Result<(), EfiError> { - if self.lock().reserved_range.is_some() { - Err(EfiError::AlreadyStarted)?; - } - - // Even though the platform is telling us what the memory buckets are, we have to take into account - // architecture-specific requirements for runtime page allocation granularity. - let granularity = self.lock().page_allocation_granularity; - - // Ensure that the requested number of pages is a multiple of the granularity - let required_pages = align_up(pages, uefi_size_to_pages!(granularity))?; - - let reserved_block_len = uefi_pages_to_size!(required_pages); + /// If the bin is full or no bin exists, the allocation falls through to the original strategy. + fn allocate_from_gcd( + &self, + strategy: AllocationStrategy, + align_shift: usize, + size: usize, + ) -> Result { + // Determine whether to attempt bin-preference allocation. + let try_bin = match strategy { + AllocationStrategy::TopDown(None) | AllocationStrategy::BottomUp(None) => true, + AllocationStrategy::TopDown(Some(max)) | AllocationStrategy::BottomUp(Some(max)) => { + if let Some(ref reserved) = self.lock().reserved_range { + max as u64 >= reserved.start + size as u64 + } else { + false + } + } + _ => false, + }; - // Allocate then free a block of the requested length in the GCD while preserving ownership. - // This, in effect, reserves this region in the GCD for use by this allocator. - let reserved_block_addr = self.gcd.allocate_memory_space( - DEFAULT_ALLOCATION_STRATEGY, - GcdMemoryType::SystemMemory, - page_shift_from_alignment(granularity)?, - reserved_block_len, - self.handle, - None, - )?; - self.gcd.free_memory_space_preserving_ownership(reserved_block_addr, reserved_block_len)?; + if try_bin + && let Some(ref reserved) = self.lock().reserved_range + && let Ok(addr) = self.gcd.allocate_memory_space( + AllocationStrategy::TopDown(Some(reserved.end as usize)), + GcdMemoryType::SystemMemory, + align_shift, + size, + self.handle, + None, + ) + { + if addr >= reserved.start as usize { + return Ok(addr); + } + // Landed below the bin, free and fall through. + let _ = self.gcd.free_memory_space(addr, size); + } - self.lock().set_reserved_range(NonNull::slice_from_raw_parts( - NonNull::new(reserved_block_addr as *mut u8).ok_or(EfiError::OutOfResources)?, - reserved_block_len, - )) + // Normal allocation path. + self.gcd.allocate_memory_space(strategy, GcdMemoryType::SystemMemory, align_shift, size, self.handle, None) } /// Returns an iterator of the ranges of memory owned by this allocator @@ -765,17 +721,13 @@ unsafe impl Allocator for SpinLockedFixedSizeBlockAllocator { // Allocate additional memory through the GCD, returning AllocError // if the GCD returns an error let start_address: usize = self - .gcd - .allocate_memory_space( + .allocate_from_gcd( DEFAULT_ALLOCATION_STRATEGY, - GcdMemoryType::SystemMemory, page_shift_from_alignment(required_alignment).map_err(|_| { debug_assert!(false); AllocError })?, allocation_size, - self.handle, - None, ) .map_err(|err| { log::error!( @@ -848,8 +800,8 @@ impl PageAllocator for SpinLockedFixedSizeBlockAllocator { unsafe { Self::free_pages(self, address, pages) } } - fn reserve_memory_pages(&self, pages: usize) -> Result<(), EfiError> { - Self::reserve_memory_pages(self, pages) + fn set_reserved_range(&self, range: Range) { + Self::set_reserved_range(self, range) } fn get_memory_ranges(&self) -> alloc::vec::IntoIter> { @@ -893,7 +845,7 @@ mod tests { use std::alloc::System; use patina::{ - base::{SIZE_64KB, UEFI_PAGE_SIZE}, + base::{SIZE_64KB, UEFI_PAGE_SHIFT, UEFI_PAGE_SIZE}, uefi_pages_to_size, }; @@ -915,12 +867,6 @@ mod tests { base } - // Test function to create a memory type info structure. - fn memory_type_info(memory_type: efi::MemoryType) -> NonNull { - let memory_type_info = Box::new(EFiMemoryTypeInformation { memory_type, number_of_pages: 0 }); - NonNull::new(Box::leak(memory_type_info)).unwrap() - } - // this runs each test twice, once with 4KB page allocation granularity and once with 64KB page allocation // granularity. This is to ensure that the allocator works correctly with both page allocation granularities. fn with_granularity_modulation(f: F) { @@ -944,7 +890,7 @@ mod tests { init_gcd(&GCD, 0x400000); - let mut fsb = FixedSizeBlockAllocator::new(memory_type_info(efi::BOOT_SERVICES_DATA), granularity); + let mut fsb = FixedSizeBlockAllocator::new(efi::BOOT_SERVICES_DATA, granularity); assert_eq!(fsb.get_memory_ranges().count(), 0); @@ -990,7 +936,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, DUMMY_HANDLE, - memory_type_info(efi::RUNTIME_SERVICES_DATA), + efi::RUNTIME_SERVICES_DATA, granularity, DEFAULT_PAGE_ALLOCATION_GRANULARITY, ); @@ -1034,10 +980,7 @@ mod tests { #[test] fn test_construct_empty_fixed_size_block_allocator() { with_locked_state(|| { - let fsb = FixedSizeBlockAllocator::new( - memory_type_info(efi::BOOT_SERVICES_DATA), - DEFAULT_PAGE_ALLOCATION_GRANULARITY, - ); + let fsb = FixedSizeBlockAllocator::new(efi::BOOT_SERVICES_DATA, DEFAULT_PAGE_ALLOCATION_GRANULARITY); assert!(fsb.list_heads.iter().all(|x| x.is_none())); assert!(fsb.allocators.is_none()); }); @@ -1054,7 +997,7 @@ mod tests { let base = init_gcd(&GCD, 0x4000000); //verify no allocators exist before expand. - let mut fsb = FixedSizeBlockAllocator::new(memory_type_info(efi::RUNTIME_SERVICES_DATA), granularity); + let mut fsb = FixedSizeBlockAllocator::new(efi::RUNTIME_SERVICES_DATA, granularity); assert!(fsb.allocators.is_none()); let allocation_size = DEFAULT_PAGE_ALLOCATION_GRANULARITY; @@ -1133,10 +1076,7 @@ mod tests { // Allocate some space on the heap with the global allocator (std) to be used by expand(). init_gcd(&GCD, 0x800000); - let mut fsb = FixedSizeBlockAllocator::new( - memory_type_info(efi::BOOT_SERVICES_DATA), - DEFAULT_PAGE_ALLOCATION_GRANULARITY, - ); + let mut fsb = FixedSizeBlockAllocator::new(efi::BOOT_SERVICES_DATA, DEFAULT_PAGE_ALLOCATION_GRANULARITY); const NUM_ALLOCATIONS: usize = 5; @@ -1179,7 +1119,7 @@ mod tests { // Allocate some space on the heap with the global allocator (std) to be used by expand(). let _ = init_gcd(&GCD, 0x400000); - let mut fsb = FixedSizeBlockAllocator::new(memory_type_info(efi::RUNTIME_SERVICES_DATA), granularity); + let mut fsb = FixedSizeBlockAllocator::new(efi::RUNTIME_SERVICES_DATA, granularity); // Test fallback_alloc with size < size_of::() let allocation_size = size_of::() / 2; @@ -1237,7 +1177,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - memory_type_info(efi::RUNTIME_SERVICES_DATA), + efi::RUNTIME_SERVICES_DATA, granularity, DEFAULT_PAGE_ALLOCATION_GRANULARITY, ); @@ -1265,7 +1205,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - memory_type_info(efi::RUNTIME_SERVICES_DATA), + efi::RUNTIME_SERVICES_DATA, granularity, DEFAULT_PAGE_ALLOCATION_GRANULARITY, ); @@ -1289,7 +1229,7 @@ mod tests { // Allocate some space on the heap with the global allocator (std) to be used by expand(). init_gcd(&GCD, 0x400000); - let mut fsb = FixedSizeBlockAllocator::new(memory_type_info(efi::RUNTIME_SERVICES_DATA), granularity); + let mut fsb = FixedSizeBlockAllocator::new(efi::RUNTIME_SERVICES_DATA, granularity); let layout = Layout::from_size_align(0x8, 0x8).unwrap(); @@ -1338,7 +1278,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - memory_type_info(efi::RUNTIME_SERVICES_DATA), + efi::RUNTIME_SERVICES_DATA, granularity, DEFAULT_PAGE_ALLOCATION_GRANULARITY, ); @@ -1379,7 +1319,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - memory_type_info(efi::RUNTIME_SERVICES_DATA), + efi::RUNTIME_SERVICES_DATA, granularity, DEFAULT_PAGE_ALLOCATION_GRANULARITY, ); @@ -1419,7 +1359,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - memory_type_info(efi::BOOT_SERVICES_DATA), + efi::BOOT_SERVICES_DATA, DEFAULT_PAGE_ALLOCATION_GRANULARITY, DEFAULT_PAGE_ALLOCATION_GRANULARITY, ); @@ -1443,7 +1383,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - memory_type_info(efi::RUNTIME_SERVICES_DATA), + efi::RUNTIME_SERVICES_DATA, granularity, DEFAULT_PAGE_ALLOCATION_GRANULARITY, ); @@ -1485,7 +1425,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - memory_type_info(efi::RUNTIME_SERVICES_DATA), + efi::RUNTIME_SERVICES_DATA, granularity, DEFAULT_PAGE_ALLOCATION_GRANULARITY, ); @@ -1522,7 +1462,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - memory_type_info(efi::RUNTIME_SERVICES_DATA), + efi::RUNTIME_SERVICES_DATA, granularity, DEFAULT_PAGE_ALLOCATION_GRANULARITY, ); @@ -1557,7 +1497,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - memory_type_info(efi::RUNTIME_SERVICES_DATA), + efi::RUNTIME_SERVICES_DATA, granularity, DEFAULT_PAGE_ALLOCATION_GRANULARITY, ); @@ -1592,7 +1532,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 0 as _, - memory_type_info(efi::BOOT_SERVICES_DATA), + efi::BOOT_SERVICES_DATA, DEFAULT_PAGE_ALLOCATION_GRANULARITY, DEFAULT_PAGE_ALLOCATION_GRANULARITY, ); @@ -1604,7 +1544,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - memory_type_info(efi::BOOT_SERVICES_DATA), + efi::BOOT_SERVICES_DATA, DEFAULT_PAGE_ALLOCATION_GRANULARITY, DEFAULT_PAGE_ALLOCATION_GRANULARITY, ); @@ -1643,7 +1583,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - memory_type_info(efi::BOOT_SERVICES_DATA), + efi::BOOT_SERVICES_DATA, DEFAULT_PAGE_ALLOCATION_GRANULARITY, DEFAULT_PAGE_ALLOCATION_GRANULARITY, ); @@ -1672,7 +1612,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - memory_type_info(efi::BOOT_SERVICES_DATA), + efi::BOOT_SERVICES_DATA, DEFAULT_PAGE_ALLOCATION_GRANULARITY, HIGH_TRAFFIC_ALLOC_MIN_EXPANSION, ); @@ -1686,19 +1626,7 @@ mod tests { assert_eq!(stats.reserved_used, 0); assert_eq!(stats.claimed_pages, 0); - //reserve some space and check the stats. - fsb.reserve_memory_pages(uefi_size_to_pages!(TEST_MIN_EXPANSION_SIZE * 2)).unwrap(); - - let stats = fsb.stats(); - assert_eq!(stats.pool_allocation_calls, 0); - assert_eq!(stats.pool_free_calls, 0); - assert_eq!(stats.page_allocation_calls, 0); - assert_eq!(stats.page_free_calls, 0); - assert_eq!(stats.reserved_size, TEST_MIN_EXPANSION_SIZE * 2); - assert_eq!(stats.reserved_used, 0); - assert_eq!(stats.claimed_pages, uefi_size_to_pages!(TEST_MIN_EXPANSION_SIZE * 2)); - - //test alloc/deallocate and stats within the bucket + //test alloc/deallocate and stats // SAFETY: fsb is initialized and used with a valid layout for testing. let ptr = unsafe { fsb.alloc( @@ -1712,9 +1640,9 @@ mod tests { assert_eq!(stats.pool_free_calls, 0); assert_eq!(stats.page_allocation_calls, 0); assert_eq!(stats.page_free_calls, 0); - assert_eq!(stats.reserved_size, TEST_MIN_EXPANSION_SIZE * 2); - assert_eq!(stats.reserved_used, TEST_MIN_EXPANSION_SIZE + uefi_pages_to_size!(1)); - assert_eq!(stats.claimed_pages, uefi_size_to_pages!(TEST_MIN_EXPANSION_SIZE * 2)); + assert_eq!(stats.reserved_size, 0); + assert_eq!(stats.reserved_used, 0); + let initial_claimed = stats.claimed_pages; // SAFETY: Allocation was returned by fsb for this layout. unsafe { @@ -1727,9 +1655,9 @@ mod tests { assert_eq!(stats.pool_free_calls, 1); assert_eq!(stats.page_allocation_calls, 0); assert_eq!(stats.page_free_calls, 0); - assert_eq!(stats.reserved_size, TEST_MIN_EXPANSION_SIZE * 2); - assert_eq!(stats.reserved_used, TEST_MIN_EXPANSION_SIZE + uefi_pages_to_size!(1)); - assert_eq!(stats.claimed_pages, uefi_size_to_pages!(TEST_MIN_EXPANSION_SIZE * 2)); + assert_eq!(stats.reserved_size, 0); + assert_eq!(stats.reserved_used, 0); + assert_eq!(stats.claimed_pages, initial_claimed); //test alloc/deallocate and stats blowing the bucket // SAFETY: fsb is initialized and used with a valid layout in tests. @@ -1747,9 +1675,9 @@ mod tests { assert_eq!(stats.pool_free_calls, 1); assert_eq!(stats.page_allocation_calls, 0); assert_eq!(stats.page_free_calls, 0); - assert_eq!(stats.reserved_size, TEST_MIN_EXPANSION_SIZE * 2); - assert_eq!(stats.reserved_used, TEST_MIN_EXPANSION_SIZE + uefi_pages_to_size!(1)); - assert_eq!(stats.claimed_pages, uefi_size_to_pages!(TEST_MIN_EXPANSION_SIZE * 5) + 1); + assert_eq!(stats.reserved_size, 0); + assert_eq!(stats.reserved_used, 0); + let claimed_after_3mb = stats.claimed_pages; // SAFETY: Allocation was returned by fsb for this layout. unsafe { @@ -1767,9 +1695,9 @@ mod tests { assert_eq!(stats.pool_free_calls, 2); assert_eq!(stats.page_allocation_calls, 0); assert_eq!(stats.page_free_calls, 0); - assert_eq!(stats.reserved_size, TEST_MIN_EXPANSION_SIZE * 2); - assert_eq!(stats.reserved_used, TEST_MIN_EXPANSION_SIZE + uefi_pages_to_size!(1)); - assert_eq!(stats.claimed_pages, uefi_size_to_pages!(TEST_MIN_EXPANSION_SIZE * 5) + 1); + assert_eq!(stats.reserved_size, 0); + assert_eq!(stats.reserved_used, 0); + assert_eq!(stats.claimed_pages, claimed_after_3mb); // test that a small page allocation fits in the 1MB free reserved region. let ptr = fsb.allocate_pages(DEFAULT_ALLOCATION_STRATEGY, 0x4, UEFI_PAGE_SIZE).unwrap().as_ptr(); @@ -1786,9 +1714,9 @@ mod tests { assert_eq!(stats.pool_free_calls, 2); assert_eq!(stats.page_allocation_calls, 1); assert_eq!(stats.page_free_calls, 0); - assert_eq!(stats.reserved_size, TEST_MIN_EXPANSION_SIZE * 2); - assert_eq!(stats.reserved_used, TEST_MIN_EXPANSION_SIZE + uefi_pages_to_size!(5)); - assert_eq!(stats.claimed_pages, uefi_size_to_pages!(TEST_MIN_EXPANSION_SIZE * 5) + 1); + assert_eq!(stats.reserved_size, 0); + assert_eq!(stats.reserved_used, 0); + assert_eq!(stats.claimed_pages, claimed_after_3mb + 0x4); // SAFETY: free_pages uses a valid test allocation pointer and page count. unsafe { @@ -1806,9 +1734,9 @@ mod tests { assert_eq!(stats.pool_free_calls, 2); assert_eq!(stats.page_allocation_calls, 1); assert_eq!(stats.page_free_calls, 1); - assert_eq!(stats.reserved_size, TEST_MIN_EXPANSION_SIZE * 2); - assert_eq!(stats.reserved_used, TEST_MIN_EXPANSION_SIZE + uefi_pages_to_size!(1)); - assert_eq!(stats.claimed_pages, uefi_size_to_pages!(TEST_MIN_EXPANSION_SIZE * 5) + 1); + assert_eq!(stats.reserved_size, 0); + assert_eq!(stats.reserved_used, 0); + assert_eq!(stats.claimed_pages, claimed_after_3mb); //test that a lage page allocation results in more claimed pages. let ptr = fsb.allocate_pages(DEFAULT_ALLOCATION_STRATEGY, 0x104, UEFI_PAGE_SIZE).unwrap().as_ptr(); @@ -1825,9 +1753,9 @@ mod tests { assert_eq!(stats.pool_free_calls, 2); assert_eq!(stats.page_allocation_calls, 2); assert_eq!(stats.page_free_calls, 1); - assert_eq!(stats.reserved_size, TEST_MIN_EXPANSION_SIZE * 2); - assert_eq!(stats.reserved_used, TEST_MIN_EXPANSION_SIZE + uefi_pages_to_size!(1)); - assert_eq!(stats.claimed_pages, uefi_size_to_pages!(TEST_MIN_EXPANSION_SIZE * 5) + 1 + 0x104); + assert_eq!(stats.reserved_size, 0); + assert_eq!(stats.reserved_used, 0); + assert_eq!(stats.claimed_pages, claimed_after_3mb + 0x104); // test that a small page allocation fits in the 1MB free reserved region. let ptr1 = fsb.allocate_pages(DEFAULT_ALLOCATION_STRATEGY, 0x4, UEFI_PAGE_SIZE).unwrap().as_ptr(); @@ -1845,9 +1773,9 @@ mod tests { assert_eq!(stats.pool_free_calls, 2); assert_eq!(stats.page_allocation_calls, 3); assert_eq!(stats.page_free_calls, 1); - assert_eq!(stats.reserved_size, TEST_MIN_EXPANSION_SIZE * 2); - assert_eq!(stats.reserved_used, TEST_MIN_EXPANSION_SIZE + uefi_pages_to_size!(5)); - assert_eq!(stats.claimed_pages, uefi_size_to_pages!(TEST_MIN_EXPANSION_SIZE * 5) + 1 + 0x104); + assert_eq!(stats.reserved_size, 0); + assert_eq!(stats.reserved_used, 0); + assert_eq!(stats.claimed_pages, claimed_after_3mb + 0x104 + 0x4); // SAFETY: free_pages uses a valid test allocation pointer and page count. unsafe { @@ -1869,9 +1797,9 @@ mod tests { assert_eq!(stats.pool_free_calls, 2); assert_eq!(stats.page_allocation_calls, 3); assert_eq!(stats.page_free_calls, 3); - assert_eq!(stats.reserved_size, TEST_MIN_EXPANSION_SIZE * 2); - assert_eq!(stats.reserved_used, TEST_MIN_EXPANSION_SIZE + uefi_pages_to_size!(1)); - assert_eq!(stats.claimed_pages, uefi_size_to_pages!(TEST_MIN_EXPANSION_SIZE * 5) + 1); + assert_eq!(stats.reserved_size, 0); + assert_eq!(stats.reserved_used, 0); + assert_eq!(stats.claimed_pages, claimed_after_3mb); }); } @@ -1885,7 +1813,7 @@ mod tests { // Allocate some space on the heap with the global allocator (std) to be used by expand(). let base = init_gcd(&GCD, 0x400000); - let mut fsb = FixedSizeBlockAllocator::new(memory_type_info(efi::RUNTIME_SERVICES_DATA), granularity); + let mut fsb = FixedSizeBlockAllocator::new(efi::RUNTIME_SERVICES_DATA, granularity); const NUM_ALLOCATIONS: usize = 3; diff --git a/patina_dxe_core/src/allocator/uefi_allocator.rs b/patina_dxe_core/src/allocator/uefi_allocator.rs index 89acb2083..e391417d8 100644 --- a/patina_dxe_core/src/allocator/uefi_allocator.rs +++ b/patina_dxe_core/src/allocator/uefi_allocator.rs @@ -70,23 +70,10 @@ where self.memory_type } - /// Reserves a range of memory to be used by this allocator of the given size in pages. - /// - /// The caller specifies a maximum number of pages this allocator is expected to require, and as long as the number - /// of pages actually used by the allocator is less than that amount, then all the allocations for this allocator - /// will be in a single contiguous block. This capability can be used to ensure that the memory map presented to the - /// OS is stable from boot-to-boot despite small boot-to-boot variations in actual page usage. - /// - /// For best memory stability, this routine should be called only during the initialization of the memory subsystem; - /// calling it after other allocations/frees have occurred will not cause allocation errors, but may cause the - /// memory map to vary from boot-to-boot. - /// - /// This routine will return Err(efi::Status::ALREADY_STARTED) if it is called more than once. - /// - pub fn reserve_memory_pages(&self, pages: usize) -> Result<(), EfiError> { - self.allocator.reserve_memory_pages(pages) + /// Sets the reserved memory range (bin range) for this allocator. + pub fn set_reserved_range(&self, range: Range) { + self.allocator.set_reserved_range(range); } - /// Returns an iterator over the memory ranges managed by this allocator. /// Returns an empty iterator if the allocator has no memory ranges. pub(crate) fn get_memory_ranges(&self) -> impl Iterator> { @@ -286,8 +273,9 @@ mod tests { use std::alloc::{GlobalAlloc, System}; use patina::{ - base::{SIZE_4KB, SIZE_64KB, UEFI_PAGE_SIZE}, + base::{SIZE_4KB, SIZE_64KB, UEFI_PAGE_SIZE, align_up, page_shift_from_alignment}, pi::dxe_services, + uefi_pages_to_size, uefi_size_to_pages, }; use crate::{ @@ -346,7 +334,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - NonNull::from_ref(GCD.memory_type_info(efi::BOOT_SERVICES_DATA)), + efi::BOOT_SERVICES_DATA, DEFAULT_PAGE_ALLOCATION_GRANULARITY, HIGH_TRAFFIC_ALLOC_MIN_EXPANSION, ); @@ -366,7 +354,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - NonNull::from_ref(GCD.memory_type_info(efi::RUNTIME_SERVICES_DATA)), + efi::RUNTIME_SERVICES_DATA, granularity, LOW_TRAFFIC_RUNTIME_ALLOC_MIN_EXPANSION, ); @@ -408,7 +396,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - NonNull::from_ref(GCD.memory_type_info(efi::RUNTIME_SERVICES_DATA)), + efi::RUNTIME_SERVICES_DATA, granularity, LOW_TRAFFIC_RUNTIME_ALLOC_MIN_EXPANSION, ); @@ -456,7 +444,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - NonNull::from_ref(GCD.memory_type_info(efi::RUNTIME_SERVICES_DATA)), + efi::RUNTIME_SERVICES_DATA, granularity, LOW_TRAFFIC_RUNTIME_ALLOC_MIN_EXPANSION, ); @@ -499,7 +487,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - NonNull::from_ref(GCD.memory_type_info(efi::RUNTIME_SERVICES_DATA)), + efi::RUNTIME_SERVICES_DATA, granularity, LOW_TRAFFIC_RUNTIME_ALLOC_MIN_EXPANSION, ); @@ -541,7 +529,7 @@ mod tests { let bs_fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - NonNull::from_ref(GCD.memory_type_info(efi::BOOT_SERVICES_DATA)), + efi::BOOT_SERVICES_DATA, DEFAULT_PAGE_ALLOCATION_GRANULARITY, HIGH_TRAFFIC_ALLOC_MIN_EXPANSION, ); @@ -550,7 +538,7 @@ mod tests { let bc_fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 2 as _, - NonNull::from_ref(GCD.memory_type_info(efi::BOOT_SERVICES_CODE)), + efi::BOOT_SERVICES_CODE, DEFAULT_PAGE_ALLOCATION_GRANULARITY, LOW_TRAFFIC_ALLOC_MIN_EXPANSION, ); @@ -583,7 +571,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - NonNull::from_ref(GCD.memory_type_info(efi::RUNTIME_SERVICES_DATA)), + efi::RUNTIME_SERVICES_DATA, granularity, LOW_TRAFFIC_RUNTIME_ALLOC_MIN_EXPANSION, ); @@ -617,7 +605,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - NonNull::from_ref(GCD.memory_type_info(efi::BOOT_SERVICES_DATA)), + efi::BOOT_SERVICES_DATA, DEFAULT_PAGE_ALLOCATION_GRANULARITY, HIGH_TRAFFIC_ALLOC_MIN_EXPANSION, ); @@ -641,7 +629,7 @@ mod tests { let fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - NonNull::from_ref(GCD.memory_type_info(efi::BOOT_SERVICES_DATA)), + efi::BOOT_SERVICES_DATA, DEFAULT_PAGE_ALLOCATION_GRANULARITY, HIGH_TRAFFIC_ALLOC_MIN_EXPANSION, ); @@ -669,6 +657,31 @@ mod tests { }); } + /// Allocates a GCD region with ownership preservation and sets it as the allocator's reserved range. + fn setup_reserved_range( + gcd: &SpinLockedGcd, + allocator: &UefiAllocator, + pages: usize, + granularity: usize, + ) -> Range { + let required_pages = align_up(pages, uefi_size_to_pages!(granularity)).unwrap(); + let size = uefi_pages_to_size!(required_pages); + let addr = gcd + .allocate_memory_space( + DEFAULT_ALLOCATION_STRATEGY, + dxe_services::GcdMemoryType::SystemMemory, + page_shift_from_alignment(granularity).unwrap(), + size, + allocator.handle(), + None, + ) + .unwrap(); + gcd.free_memory_space_preserving_ownership(addr, size).unwrap(); + let range = addr as efi::PhysicalAddress..(addr + size) as efi::PhysicalAddress; + allocator.set_reserved_range(range.clone()); + range + } + #[test] fn reserve_memory_pages_reserves_the_pages() { with_granularity_modulation(|granularity| { @@ -681,17 +694,17 @@ mod tests { let reserved_fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 1 as _, - NonNull::from_ref(GCD.memory_type_info(efi::RUNTIME_SERVICES_DATA)), + efi::RUNTIME_SERVICES_DATA, granularity, LOW_TRAFFIC_RUNTIME_ALLOC_MIN_EXPANSION, ); let reserved_allocator = UefiAllocator::new(reserved_fsb, efi::RUNTIME_SERVICES_DATA); - reserved_allocator.reserve_memory_pages(0x100).unwrap(); + setup_reserved_range(&GCD, &reserved_allocator, 0x100, granularity); let unreserved_fsb = SpinLockedFixedSizeBlockAllocator::new( &GCD, 2 as _, - NonNull::from_ref(GCD.memory_type_info(efi::LOADER_DATA)), + efi::LOADER_DATA, DEFAULT_PAGE_ALLOCATION_GRANULARITY, LOW_TRAFFIC_ALLOC_MIN_EXPANSION, ); @@ -783,4 +796,46 @@ mod tests { }); }); } + + #[test] + fn allocate_max_address_prefers_reserved_range() { + with_granularity_modulation(|granularity| { + with_locked_state(|| { + static GCD: SpinLockedGcd = SpinLockedGcd::new(None); + + let base = init_gcd(&GCD, 0x400000); + let gcd_end = base + 0x400000; + + let reserved_fsb = SpinLockedFixedSizeBlockAllocator::new( + &GCD, + 1 as _, + efi::RUNTIME_SERVICES_DATA, + granularity, + LOW_TRAFFIC_RUNTIME_ALLOC_MIN_EXPANSION, + ); + let reserved_allocator = UefiAllocator::new(reserved_fsb, efi::RUNTIME_SERVICES_DATA); + let reserved_range = setup_reserved_range(&GCD, &reserved_allocator, 0x100, granularity); + + // TopDown(Some(max)) where max is above the reserved range should land in the bin. + let page = reserved_allocator + .allocate_pages(AllocationStrategy::TopDown(Some(gcd_end as usize)), 1, UEFI_PAGE_SIZE) + .unwrap(); + let page_addr = page.as_ptr() as *mut u8 as u64; + assert!( + reserved_range.contains(&page_addr), + "TopDown(Some(gcd_end)) should land in the bin: addr={page_addr:#x}, range={reserved_range:#x?}", + ); + + // TopDown(Some(max)) where max is below the reserved range must not try the bin. + let page_below = reserved_allocator + .allocate_pages(AllocationStrategy::TopDown(Some(reserved_range.start as usize)), 1, UEFI_PAGE_SIZE) + .unwrap(); + let page_below_addr = page_below.as_ptr() as *mut u8 as u64; + assert!( + !reserved_range.contains(&page_below_addr), + "TopDown(Some(below_bin)) should not land in the bin: addr={page_below_addr:#x}, range={reserved_range:#x?}", + ); + }); + }); + } } diff --git a/patina_dxe_core/src/gcd.rs b/patina_dxe_core/src/gcd.rs index a6c23dcb5..f764f4e78 100644 --- a/patina_dxe_core/src/gcd.rs +++ b/patina_dxe_core/src/gcd.rs @@ -21,7 +21,7 @@ use patina::{ error::EfiError, pi::{ dxe_services::{GcdIoType, GcdMemoryType, MemorySpaceDescriptor}, - hob::{self, Hob, HobList, PhaseHandoffInformationTable}, + hob::{self, Hob, HobList, MEMORY_TYPE_INFO_HOB_GUID, PhaseHandoffInformationTable}, }, }; use patina_internal_cpu::paging::{PatinaPageTable, create_cpu_paging}; @@ -557,6 +557,12 @@ pub fn add_hob_resource_descriptors_to_gcd(hob_list: &HobList) { None => continue, // Not a resource descriptor HOB or unsupported version for this build }; + // Skip the PEI memory bin region to avoid a conflict. It will overlap the system + // memory region it was allocated from. + if res_desc.owner == MEMORY_TYPE_INFO_HOB_GUID { + continue; + } + let mem_range = res_desc.physical_start ..res_desc.physical_start.checked_add(res_desc.resource_length).expect("Invalid resource descriptor hob"); diff --git a/patina_dxe_core/src/gcd/spin_locked_gcd.rs b/patina_dxe_core/src/gcd/spin_locked_gcd.rs index 98de2b873..fbf8984e8 100644 --- a/patina_dxe_core/src/gcd/spin_locked_gcd.rs +++ b/patina_dxe_core/src/gcd/spin_locked_gcd.rs @@ -17,7 +17,7 @@ use patina::{ guids::{self, CACHE_ATTRIBUTE_CHANGE_EVENT_GROUP}, pi::{ dxe_services::{self, GcdMemoryType, MemorySpaceDescriptor}, - hob::{self, EFiMemoryTypeInformation}, + hob, }, uefi_pages_to_size, uefi_size_to_pages, }; @@ -1971,7 +1971,6 @@ pub struct SpinLockedGcd { memory: tpl_mutex::TplMutex, io: tpl_mutex::TplMutex, memory_change_callback: Option, - memory_type_info_table: [EFiMemoryTypeInformation; 17], page_table: tpl_mutex::TplMutex>>, /// Contains the current memory protection policy pub(crate) memory_protection_policy: MemoryProtectionPolicy, @@ -2007,25 +2006,6 @@ impl SpinLockedGcd { "GcdIoLock", ), memory_change_callback, - memory_type_info_table: [ - EFiMemoryTypeInformation { memory_type: efi::RESERVED_MEMORY_TYPE, number_of_pages: 0 }, - EFiMemoryTypeInformation { memory_type: efi::LOADER_CODE, number_of_pages: 0 }, - EFiMemoryTypeInformation { memory_type: efi::LOADER_DATA, number_of_pages: 0 }, - EFiMemoryTypeInformation { memory_type: efi::BOOT_SERVICES_CODE, number_of_pages: 0 }, - EFiMemoryTypeInformation { memory_type: efi::BOOT_SERVICES_DATA, number_of_pages: 0 }, - EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_CODE, number_of_pages: 0 }, - EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: 0 }, - EFiMemoryTypeInformation { memory_type: efi::CONVENTIONAL_MEMORY, number_of_pages: 0 }, - EFiMemoryTypeInformation { memory_type: efi::UNUSABLE_MEMORY, number_of_pages: 0 }, - EFiMemoryTypeInformation { memory_type: efi::ACPI_RECLAIM_MEMORY, number_of_pages: 0 }, - EFiMemoryTypeInformation { memory_type: efi::ACPI_MEMORY_NVS, number_of_pages: 0 }, - EFiMemoryTypeInformation { memory_type: efi::MEMORY_MAPPED_IO, number_of_pages: 0 }, - EFiMemoryTypeInformation { memory_type: efi::MEMORY_MAPPED_IO_PORT_SPACE, number_of_pages: 0 }, - EFiMemoryTypeInformation { memory_type: efi::PAL_CODE, number_of_pages: 0 }, - EFiMemoryTypeInformation { memory_type: efi::PERSISTENT_MEMORY, number_of_pages: 0 }, - EFiMemoryTypeInformation { memory_type: efi::UNACCEPTED_MEMORY_TYPE, number_of_pages: 0 }, - EFiMemoryTypeInformation { memory_type: 16 /*EfiMaxMemoryType*/, number_of_pages: 0 }, - ], page_table: tpl_mutex::TplMutex::new(efi::TPL_HIGH_LEVEL, None, "GcdPageTableLock"), memory_protection_policy: MemoryProtectionPolicy::new(), last_efi_memory_map_key: tpl_mutex::TplMutex::new(efi::TPL_HIGH_LEVEL, None, "LastEfiMemoryMapKeyLock"), @@ -2055,16 +2035,6 @@ impl SpinLockedGcd { self.memory.lock().prioritize_32_bit_memory = value; } - /// Returns a reference to the memory type information table. - pub const fn memory_type_info_table(&self) -> &[EFiMemoryTypeInformation; 17] { - &self.memory_type_info_table - } - - /// Returns a pointer to the memory type information for the given memory type. - pub const fn memory_type_info(&self, memory_type: u32) -> &EFiMemoryTypeInformation { - &self.memory_type_info_table[memory_type as usize] - } - fn set_paging_attributes(&self, base_address: usize, len: usize, attributes: u64) -> Result<(), EfiError> { if let Some(page_table) = &mut *self.page_table.lock() { // only apply page table attributes to the page table, not our virtual GCD attributes diff --git a/patina_dxe_core/src/lib.rs b/patina_dxe_core/src/lib.rs index 26d162e90..2dfd2b5c1 100644 --- a/patina_dxe_core/src/lib.rs +++ b/patina_dxe_core/src/lib.rs @@ -84,6 +84,7 @@ mod events; mod filesystems; mod gcd; mod memory_attributes_protocol; +mod memory_bin; mod memory_manager; mod misc_boot_services; mod pecoff; @@ -428,6 +429,7 @@ impl Core

{ //make sure that well-known handles exist. PROTOCOL_DB.init_protocol_db(); + // Initialize full allocation support. allocator::init_memory_support(&hob_list); diff --git a/patina_dxe_core/src/memory_bin.rs b/patina_dxe_core/src/memory_bin.rs new file mode 100644 index 000000000..3c870a1ef --- /dev/null +++ b/patina_dxe_core/src/memory_bin.rs @@ -0,0 +1,1572 @@ +//! Memory Bin Manager +//! +//! Tracks memory bin regions for hibernate (S4) resume stability. A "memory bin" is +//! a pre-allocated address range for a specific EFI memory type whose size is defined +//! by the platform's Memory Type Information HOB. +//! +//! This module is responsible for: +//! +//! 1. HOB processing: Extracting bin configuration from the Memory Type Information +//! GUID HOB and optionally consuming a pre-allocated bin region from a Resource +//! Descriptor HOB produced by PEI. +//! +//! 2. GetMemoryMap "overlay": Post-processing the EFI memory map so that free +//! (`EfiConventionalMemory`) pages within a bin region are reported as the bin's +//! memory type. +//! +//! 3. Statistics and config table: Tracking per-type allocation counts and bin +//! usage so BDS can recommend bin sizes for the next boot. +//! +//! ## License +//! +//! Copyright (c) Microsoft Corporation. +//! +//! SPDX-License-Identifier: Apache-2.0 +//! + +use crate::allocator::{DEFAULT_PAGE_ALLOCATION_GRANULARITY, RUNTIME_PAGE_ALLOCATION_GRANULARITY}; +use patina::{ + base::{align_pages_to_granularity, align_up}, + efi_types::{EFI_MAX_MEMORY_TYPE, INVALID_INFORMATION_INDEX}, + pi::hob::{self, EFiMemoryTypeInformation, Hob, HobList, MEMORY_TYPE_INFO_HOB_GUID}, + uefi_pages_to_size, uefi_size_to_pages, +}; +use r_efi::efi; + +extern crate alloc; +use alloc::vec::Vec; + +/// Maximum number of entries in the memory type information array. +const MAX_MEMORY_TYPE_INFO_ENTRIES: usize = EFI_MAX_MEMORY_TYPE + 1; + +/// Maximum allocation address. +const MAX_ALLOC_ADDRESS: efi::PhysicalAddress = u64::MAX >> 1; + +/// Log target for all memory bin log messages. +const LOG_TARGET: &str = "memory_bin"; + +/// Returns a human-readable name for a UEFI memory type. +/// +/// Returns a `&'static str` for all standard types. Returns `"Unknown"` for unrecognized values. +pub(crate) fn memory_type_name(memory_type: efi::MemoryType) -> &'static str { + match memory_type { + efi::RESERVED_MEMORY_TYPE => "ReservedMemoryType", + efi::LOADER_CODE => "LoaderCode", + efi::LOADER_DATA => "LoaderData", + efi::BOOT_SERVICES_CODE => "BootServicesCode", + efi::BOOT_SERVICES_DATA => "BootServicesData", + efi::RUNTIME_SERVICES_CODE => "RuntimeServicesCode", + efi::RUNTIME_SERVICES_DATA => "RuntimeServicesData", + efi::CONVENTIONAL_MEMORY => "ConventionalMemory", + efi::UNUSABLE_MEMORY => "UnusableMemory", + efi::ACPI_RECLAIM_MEMORY => "ACPIReclaimMemory", + efi::ACPI_MEMORY_NVS => "ACPIMemoryNVS", + efi::MEMORY_MAPPED_IO => "MemoryMappedIO", + efi::MEMORY_MAPPED_IO_PORT_SPACE => "MemoryMappedIOPortSpace", + efi::PAL_CODE => "PalCode", + efi::PERSISTENT_MEMORY => "PersistentMemory", + efi::UNACCEPTED_MEMORY_TYPE => "UnacceptedMemoryType", + _ => "Unknown", + } +} + +/// Per-memory-type bin statistics. +/// +/// Tracks the bin region, current allocation count, and metadata for a single memory type. +/// Mirrors `EFI_MEMORY_TYPE_STATISTICS` in edk2. +#[derive(Debug, Clone, Copy)] +struct MemoryBinStatistics { + /// The base (lowest) address of this memory type's bin region. + base_address: efi::PhysicalAddress, + /// The maximum (highest) address of this memory type's bin region. + maximum_address: efi::PhysicalAddress, + /// The number of pages currently allocated within this bin. + current_number_of_pages: u64, + /// The total number of pages reserved for this bin. + number_of_pages: u64, + /// Index into the `MemoryTypeInformation` array for this type. + information_index: usize, + /// Whether this memory type persists into the OS runtime (affects `GetMemoryMap` behavior). + special: bool, + /// Whether this memory type should have `EFI_MEMORY_RUNTIME` attribute in the memory map. + runtime: bool, +} + +impl MemoryBinStatistics { + /// Creates default statistics for a memory type with the given special/runtime flags. + const fn new(special: bool, runtime: bool) -> Self { + Self { + base_address: 0, + maximum_address: MAX_ALLOC_ADDRESS, + current_number_of_pages: 0, + number_of_pages: 0, + information_index: INVALID_INFORMATION_INDEX, + special, + runtime, + } + } +} + +/// Default `MemoryBinStatistics` initialization for all memory types. +/// +/// Indexed by `efi::MemoryType` value. Matches edk2's `mMemoryTypeStatistics` initialization. +const DEFAULT_STATISTICS: [MemoryBinStatistics; EFI_MAX_MEMORY_TYPE + 1] = [ + MemoryBinStatistics::new(true, false), // EfiReservedMemoryType (0) + MemoryBinStatistics::new(false, false), // EfiLoaderCode (1) + MemoryBinStatistics::new(false, false), // EfiLoaderData (2) + MemoryBinStatistics::new(false, false), // EfiBootServicesCode (3) + MemoryBinStatistics::new(false, false), // EfiBootServicesData (4) + MemoryBinStatistics::new(true, true), // EfiRuntimeServicesCode (5) + MemoryBinStatistics::new(true, true), // EfiRuntimeServicesData (6) + MemoryBinStatistics::new(false, false), // EfiConventionalMemory (7) + MemoryBinStatistics::new(false, false), // EfiUnusableMemory (8) + MemoryBinStatistics::new(true, false), // EfiACPIReclaimMemory (9) + MemoryBinStatistics::new(true, false), // EfiACPIMemoryNVS (10) + MemoryBinStatistics::new(false, false), // EfiMemoryMappedIO (11) + MemoryBinStatistics::new(false, false), // EfiMemoryMappedIOPortSpace (12) + MemoryBinStatistics::new(true, true), // EfiPalCode (13) + MemoryBinStatistics::new(false, false), // EfiPersistentMemory (14) + MemoryBinStatistics::new(true, false), // EfiUnacceptedMemoryType (15) + MemoryBinStatistics::new(false, false), // EfiMaxMemoryType sentinel (16) +]; + +/// Manages memory bins for hibernate resume stability. +/// +/// The `MemoryBinManager` tracks per-memory-type bin regions and allocation statistics. +pub(crate) struct MemoryBinManager { + /// Per-memory-type bin statistics, indexed by `efi::MemoryType`. + statistics: [MemoryBinStatistics; EFI_MAX_MEMORY_TYPE + 1], + /// Current memory type information with peak usage tracking for the BDS config table. + /// This is a fixed-size array so that raw pointers to it remain valid for the + /// lifetime of the static `MEMORY_BIN_MANAGER`. + memory_type_information: [EFiMemoryTypeInformation; MAX_MEMORY_TYPE_INFO_ENTRIES], + /// Number of valid entries in `memory_type_information`. + memory_type_information_count: usize, + /// Whether bins have been initialized. + initialized: bool, +} + +impl MemoryBinManager { + /// Creates a new uninitialized `MemoryBinManager`. + pub(crate) const fn new() -> Self { + Self { + statistics: DEFAULT_STATISTICS, + memory_type_information: [EFiMemoryTypeInformation { memory_type: 0, number_of_pages: 0 }; + MAX_MEMORY_TYPE_INFO_ENTRIES], + memory_type_information_count: 0, + initialized: false, + } + } + + /// Returns whether memory bins have been initialized. + pub(crate) fn is_initialized(&self) -> bool { + self.initialized + } + + /// Returns the allocation granularity for the given memory type. + pub const fn granularity_for_type(memory_type: efi::MemoryType) -> usize { + match memory_type { + efi::RESERVED_MEMORY_TYPE + | efi::ACPI_MEMORY_NVS + | efi::RUNTIME_SERVICES_CODE + | efi::RUNTIME_SERVICES_DATA => RUNTIME_PAGE_ALLOCATION_GRANULARITY, + _ => DEFAULT_PAGE_ALLOCATION_GRANULARITY, + } + } + + /// Calculates the total memory needed for all bins, considering alignment. + /// + /// If `bin_top` is non-zero, alignment padding is included in the calculation. + fn calculate_total_bin_size(memory_type_info: &[EFiMemoryTypeInformation], bin_top: efi::PhysicalAddress) -> u64 { + let mut total_size: u64 = 0; + let mut current_top = bin_top; + + for entry in memory_type_info { + if entry.memory_type as usize >= EFI_MAX_MEMORY_TYPE { + break; + } + + let granularity = Self::granularity_for_type(entry.memory_type) as u64; + let entry_size = uefi_pages_to_size!(entry.number_of_pages as usize) as u64; + total_size += entry_size; + + if current_top == 0 { + continue; + } + + current_top -= entry_size; + let alignment_padding = current_top & (granularity - 1); + total_size += alignment_padding; + current_top &= !(granularity - 1); + } + + total_size + } + + /// Calculates a conservative allocation size for a single contiguous bin block. + /// + /// Returns `None` if there are no bin entries with pages > 0. + /// The result includes the raw entry sizes plus worst-case per-entry alignment padding, rounded + /// up to the maximum bin granularity. + pub(crate) fn contiguous_alloc_size(memory_type_info: &[EFiMemoryTypeInformation]) -> Option { + let mut raw_total: usize = 0; + let mut entry_count: usize = 0; + let mut max_granularity = DEFAULT_PAGE_ALLOCATION_GRANULARITY; + + for entry in memory_type_info { + if entry.memory_type as usize >= EFI_MAX_MEMORY_TYPE { + break; + } + if entry.number_of_pages == 0 { + continue; + } + raw_total += uefi_pages_to_size!(entry.number_of_pages as usize); + entry_count += 1; + max_granularity = max_granularity.max(Self::granularity_for_type(entry.memory_type)); + } + + if raw_total == 0 { + return None; + } + + // Each entry may need up to (granularity - 1) bytes of alignment padding within the block. + // Using max_granularity per entry is a safe over-estimate. + let padded = raw_total + entry_count * max_granularity; + Some(align_up(padded, max_granularity).unwrap_or(padded)) + } + + /// Returns the maximum allocation granularity across all non-zero bin entries. + pub(crate) fn max_granularity(memory_type_info: &[EFiMemoryTypeInformation]) -> usize { + memory_type_info + .iter() + .take_while(|e| (e.memory_type as usize) < EFI_MAX_MEMORY_TYPE) + .filter(|e| e.number_of_pages > 0) + .map(|e| Self::granularity_for_type(e.memory_type)) + .max() + .unwrap_or(DEFAULT_PAGE_ALLOCATION_GRANULARITY) + } + + /// Initializes bins from a pre-allocated range (provided via Resource Descriptor HOB from PEI). + /// + /// Divides the range `[start, start + length)` into per-type bins based on `memory_type_info`. + /// Bins are allocated from the top of the range downward. + pub(crate) fn initialize_from_range( + &mut self, + start: efi::PhysicalAddress, + length: u64, + memory_type_info: &[EFiMemoryTypeInformation], + ) -> bool { + if self.initialized { + log::warn!("Memory bins already initialized, ignoring range."); + return false; + } + + let total_needed = Self::calculate_total_bin_size(memory_type_info, start + length); + if total_needed > length { + log::warn!( + target: LOG_TARGET, + "Memory bin range too small: need {:#X} bytes but only {:#X} available.", + total_needed, + length + ); + return false; + } + + log::info!( + target: LOG_TARGET, + "Initializing memory bins from PEI range: base={:#X} length={:#X} total_needed={:#X}", + start, + length, + total_needed + ); + + let mut top = start + length; + + for (index, entry) in memory_type_info.iter().enumerate() { + let mem_type = entry.memory_type; + if mem_type as usize >= EFI_MAX_MEMORY_TYPE { + break; + } + if entry.number_of_pages == 0 { + continue; + } + + let entry_size = uefi_pages_to_size!(entry.number_of_pages as usize) as u64; + let stats = &mut self.statistics[mem_type as usize]; + stats.maximum_address = top - 1; + top -= entry_size; + + // Align to the type's granularity + let granularity = Self::granularity_for_type(mem_type) as u64; + top &= !(granularity - 1); + + stats.base_address = top; + stats.number_of_pages = entry.number_of_pages as u64; + stats.information_index = index; + + log::info!( + target: LOG_TARGET, + " Bin[{}] {}: base={:#X} max={:#X} pages={:#X} ({} pages)", + mem_type, + memory_type_name(mem_type), + stats.base_address, + stats.maximum_address, + stats.number_of_pages, + stats.number_of_pages + ); + } + + self.finalize_information_index(memory_type_info); + self.copy_memory_type_info(memory_type_info); + self.initialized = true; + + log::info!( + target: LOG_TARGET, + "Memory bins initialized from pre-allocated range." + ); + true + } + + /// Sets the `information_index` for each memory type that has a corresponding entry + /// in the memory type information array. + fn finalize_information_index(&mut self, memory_type_info: &[EFiMemoryTypeInformation]) { + for mem_type in 0..EFI_MAX_MEMORY_TYPE { + for (index, entry) in memory_type_info.iter().enumerate() { + if mem_type == entry.memory_type as usize { + self.statistics[mem_type].information_index = index; + } + } + self.statistics[mem_type].current_number_of_pages = 0; + } + log::trace!(target: LOG_TARGET, "Bin stats: finalized information indices, reset current_number_of_pages to 0 for all types"); + } + + /// Copies memory type information entries into the fixed-size array. + fn copy_memory_type_info(&mut self, memory_type_info: &[EFiMemoryTypeInformation]) { + let count = memory_type_info.len().min(MAX_MEMORY_TYPE_INFO_ENTRIES); + self.memory_type_information[..count].copy_from_slice(&memory_type_info[..count]); + self.memory_type_information_count = count; + + if log::log_enabled!(target: LOG_TARGET, log::Level::Trace) { + log::trace!( + target: LOG_TARGET, + "Bin table: initialized with {} entries from HOB", + count + ); + + for entry in &self.memory_type_information[..count] { + log::trace!( + target: LOG_TARGET, + " Bin table: {} pages={}", + memory_type_name(entry.memory_type), + entry.number_of_pages + ); + } + } + } + + /// Seeds bin statistics from a PEI memory allocation HOB. + /// + /// Called for Memory Allocation HOBs that have `name == MEMORY_TYPE_INFO_HOB_GUID`, + /// indicating they were produced by PEI's bin-aware allocator. All memory bin-type + /// allocations are expected to be counted regardless of address. + pub(crate) fn seed_statistics_from_hob(&mut self, memory_type: efi::MemoryType, pages: u64) { + if !self.initialized { + return; + } + + let type_idx = memory_type as usize; + if type_idx >= EFI_MAX_MEMORY_TYPE || !self.statistics[type_idx].special { + return; + } + + let aligned_pages = align_pages_to_granularity(pages, Self::granularity_for_type(memory_type)); + self.statistics[type_idx].current_number_of_pages += aligned_pages; + log::debug!( + target: LOG_TARGET, + "PEI seed: {} +{} pages. total={}", + memory_type_name(memory_type), + pages, + self.statistics[type_idx].current_number_of_pages + ); + } + + /// Returns an iterator over all active bins: `(memory_type, base_address, max_address, pages)`. + /// + /// Only yields bins with `number_of_pages > 0`. + pub(crate) fn active_bins( + &self, + ) -> impl Iterator + '_ { + self.statistics.iter().enumerate().filter_map(|(idx, stats)| { + if stats.number_of_pages > 0 && idx < EFI_MAX_MEMORY_TYPE { + Some((idx as efi::MemoryType, stats.base_address, stats.maximum_address, stats.number_of_pages)) + } else { + None + } + }) + } + + /// Records an allocation for statistics tracking on special (runtime) memory types. + /// + /// Only tracks types with active bins (`special == true` and `number_of_pages > 0`). + /// Updates `current_number_of_pages` and peak tracking in `memory_type_information`. + pub(crate) fn record_allocation(&mut self, memory_type: efi::MemoryType, pages: u64) { + if !self.initialized { + return; + } + + let type_idx = memory_type as usize; + if type_idx >= EFI_MAX_MEMORY_TYPE || !self.statistics[type_idx].special { + return; + } + + let aligned_pages = align_pages_to_granularity(pages, Self::granularity_for_type(memory_type)); + let prev = self.statistics[type_idx].current_number_of_pages; + self.statistics[type_idx].current_number_of_pages += aligned_pages; + + log::trace!( + target: LOG_TARGET, + "Bin stats: {} current_pages {} -> {} (alloc +{} aligned to {})", + memory_type_name(memory_type), + prev, + self.statistics[type_idx].current_number_of_pages, + pages, + aligned_pages + ); + + // Update peak tracking: if current exceeds previous peak, update for BDS + let info_idx = self.statistics[type_idx].information_index; + if info_idx < self.memory_type_information_count + && self.statistics[type_idx].current_number_of_pages + > self.memory_type_information[info_idx].number_of_pages as u64 + { + let prev_peak = self.memory_type_information[info_idx].number_of_pages; + self.memory_type_information[info_idx].number_of_pages = + self.statistics[type_idx].current_number_of_pages as u32; + log::trace!( + target: LOG_TARGET, + "Bin table: {} pages {} -> {} (peak update)", + memory_type_name(memory_type), + prev_peak, + self.memory_type_information[info_idx].number_of_pages + ); + } + } + + /// Records a deallocation for statistics tracking on special (runtime) memory types. + /// + /// Like [`Self::record_allocation`], all special-type frees are counted regardless of address. + pub(crate) fn record_free(&mut self, memory_type: efi::MemoryType, pages: u64) { + if !self.initialized { + return; + } + + let type_idx = memory_type as usize; + if type_idx >= EFI_MAX_MEMORY_TYPE || !self.statistics[type_idx].special { + return; + } + + let aligned_pages = align_pages_to_granularity(pages, Self::granularity_for_type(memory_type)); + let prev = self.statistics[type_idx].current_number_of_pages; + self.statistics[type_idx].current_number_of_pages = + self.statistics[type_idx].current_number_of_pages.saturating_sub(aligned_pages); + + log::trace!( + target: LOG_TARGET, + "Bin stats: {} current_pages {} -> {} (free -{} aligned to {})", + memory_type_name(memory_type), + prev, + self.statistics[type_idx].current_number_of_pages, + pages, + aligned_pages + ); + } + + /// Applies bin descriptors to a populated EFI memory map buffer. + /// + /// Post-processes the memory map by converting `EfiConventionalMemory` entries that overlap + /// with bin regions to the bin's memory type. Entries may be split at bin boundaries. + /// + /// `count` is the current number of valid entries in `buffer`. + /// Returns the new entry count after splitting and conversion. + pub(crate) fn apply_bin_descriptors(&self, buffer: &mut [efi::MemoryDescriptor], count: usize) -> usize { + if !self.initialized { + return count; + } + + let mut current_count = count; + + for mem_type in 0..(EFI_MAX_MEMORY_TYPE as u32) { + let stats = &self.statistics[mem_type as usize]; + + // Only process special types with actual bin pages + if stats.number_of_pages == 0 || !stats.special { + continue; + } + + let bin_start = stats.base_address; + let bin_end = stats.maximum_address; + + log::debug!( + target: LOG_TARGET, + "GetMemoryMap: processing bin[{}] {} range=[{:#X}..{:#X}]", + mem_type, + memory_type_name(mem_type), + bin_start, + bin_end + ); + + // Repeatedly process until no more modifications are needed. + // Each pass may split one entry, so restart from the beginning after each modification. + loop { + current_count = Self::merge_descriptors(buffer, current_count); + + let entry_count = current_count; + let mut did_modify = false; + + for i in 0..entry_count { + if buffer[i].r#type != efi::CONVENTIONAL_MEMORY { + continue; + } + + let entry_start = buffer[i].physical_start; + let entry_end = entry_start + uefi_pages_to_size!(buffer[i].number_of_pages as usize) as u64 - 1; + + // No overlap + if entry_end < bin_start || entry_start > bin_end { + continue; + } + + // Case 1: Entry completely within bin + if entry_start >= bin_start && entry_end <= bin_end { + Self::set_descriptor_type(&mut buffer[i], mem_type, stats.runtime); + did_modify = true; + break; + } + + // Case 2: Entry starts before bin + if entry_start < bin_start { + if current_count + 1 >= buffer.len() { + log::error!("Buffer too small for memory bin descriptor split."); + return current_count; + } + + // Shrink original entry to end at bin start + let pre_bin_pages = uefi_size_to_pages!((bin_start - entry_start) as usize); + buffer[i].number_of_pages = pre_bin_pages as u64; + + // Insert new entry for in-bin portion + current_count = Self::insert_descriptor_after(buffer, current_count, i); + let new_idx = i + 1; + buffer[new_idx].physical_start = bin_start; + buffer[new_idx].number_of_pages = + uefi_size_to_pages!((entry_end - bin_start + 1) as usize) as u64; + Self::set_descriptor_type(&mut buffer[new_idx], mem_type, stats.runtime); + + // If entry also extends past bin end, split again + if entry_end > bin_end { + if current_count + 1 >= buffer.len() { + log::error!("Buffer too small for memory bin descriptor split."); + return current_count; + } + + buffer[new_idx].number_of_pages = + uefi_size_to_pages!((bin_end - bin_start + 1) as usize) as u64; + + current_count = Self::insert_descriptor_after(buffer, current_count, new_idx); + let post_idx = new_idx + 1; + buffer[post_idx].physical_start = bin_end + 1; + buffer[post_idx].number_of_pages = + uefi_size_to_pages!((entry_end - bin_end) as usize) as u64; + Self::set_descriptor_type(&mut buffer[post_idx], efi::CONVENTIONAL_MEMORY, false); + } + + did_modify = true; + break; + } + + // Case 3: Entry ends after bin (entry_start >= bin_start implied here) + if entry_end > bin_end { + if current_count + 1 >= buffer.len() { + log::error!("Buffer too small for memory bin descriptor split."); + return current_count; + } + + // Shrink original entry to cover only the in-bin portion + buffer[i].number_of_pages = uefi_size_to_pages!((bin_end - entry_start + 1) as usize) as u64; + Self::set_descriptor_type(&mut buffer[i], mem_type, stats.runtime); + + // Insert new entry for the post-bin portion + current_count = Self::insert_descriptor_after(buffer, current_count, i); + let post_idx = i + 1; + buffer[post_idx].physical_start = bin_end + 1; + buffer[post_idx].number_of_pages = uefi_size_to_pages!((entry_end - bin_end) as usize) as u64; + Self::set_descriptor_type(&mut buffer[post_idx], efi::CONVENTIONAL_MEMORY, false); + + did_modify = true; + break; + } + } + + if !did_modify { + break; + } + } + } + + Self::merge_descriptors(buffer, current_count) + } + + /// Returns the current memory type information for config table publishing. + /// + /// Contains peak usage data that BDS can use to recommend next-boot bin sizes. + pub(crate) fn memory_type_information(&self) -> &[EFiMemoryTypeInformation] { + &self.memory_type_information[..self.memory_type_information_count] + } + + /// Returns the maximum number of additional descriptors that bin splitting could add. + /// + /// Each active memory bin can cause up to 2 additional descriptor entries (worst case + /// where an entry spans the entire bin, requiring a triple-split). + pub(crate) fn max_additional_descriptors(&self) -> usize { + if !self.initialized { + return 0; + } + + self.statistics.iter().filter(|s| s.number_of_pages > 0 && s.special).count() * 2 + } + + /// Sets the type and runtime attribute on a memory descriptor. + fn set_descriptor_type(descriptor: &mut efi::MemoryDescriptor, memory_type: efi::MemoryType, runtime: bool) { + descriptor.r#type = memory_type; + if runtime { + descriptor.attribute |= efi::MEMORY_RUNTIME; + } else { + descriptor.attribute &= !efi::MEMORY_RUNTIME; + } + } + + /// Inserts a new descriptor after position `after_idx` by shifting subsequent entries right. + /// + /// The new entry is initialized as a copy of `buffer[after_idx]`. + /// Returns the new total count. + fn insert_descriptor_after(buffer: &mut [efi::MemoryDescriptor], count: usize, after_idx: usize) -> usize { + // Shift entries after `after_idx` right by one + buffer.copy_within((after_idx + 1)..count, after_idx + 2); + // Copy the current entry as a template for the new one + buffer[after_idx + 1] = buffer[after_idx]; + count + 1 + } + + /// Merges consecutive descriptors with the same type and attributes. + /// + /// Returns the new count after merging. + fn merge_descriptors(buffer: &mut [efi::MemoryDescriptor], count: usize) -> usize { + if count <= 1 { + return count; + } + + let mut write_idx = 0; + for read_idx in 1..count { + let prev_end = buffer[write_idx].physical_start + + uefi_pages_to_size!(buffer[write_idx].number_of_pages as usize) as u64; + + if buffer[read_idx].r#type == buffer[write_idx].r#type + && buffer[read_idx].attribute == buffer[write_idx].attribute + && buffer[read_idx].physical_start == prev_end + { + // Merge into the current entry + buffer[write_idx].number_of_pages += buffer[read_idx].number_of_pages; + } else { + write_idx += 1; + if write_idx != read_idx { + buffer[write_idx] = buffer[read_idx]; + } + } + } + + write_idx + 1 + } + + /// Resets the bin manager to its initial uninitialized state. + #[cfg(test)] + pub(crate) fn reset(&mut self) { + self.statistics = DEFAULT_STATISTICS; + self.memory_type_information = + [EFiMemoryTypeInformation { memory_type: 0, number_of_pages: 0 }; MAX_MEMORY_TYPE_INFO_ENTRIES]; + self.memory_type_information_count = 0; + self.initialized = false; + } +} + +/// Searches the HOB list for a Resource Descriptor HOB owned by `gEfiMemoryTypeInformationGuid`. +/// +/// Returns `Some((physical_start, resource_length))` if exactly one valid Resource Descriptor HOB +/// is found with the correct owner, resource type, and attributes. Returns `None` if no match +/// or multiple matches are found. +pub(crate) fn find_memory_type_info_resource_hob( + hob_list: &HobList, + memory_type_info: &[EFiMemoryTypeInformation], +) -> Option<(efi::PhysicalAddress, u64)> { + let target_guid = MEMORY_TYPE_INFO_HOB_GUID; + let mut count = 0u32; + let mut result: Option<(efi::PhysicalAddress, u64)> = None; + + for hob_entry in hob_list.iter() { + let res_desc = match hob_entry { + Hob::ResourceDescriptor(rd) => rd, + _ => continue, + }; + + if res_desc.owner != target_guid { + continue; + } + + count += 1; + + if res_desc.resource_type != hob::EFI_RESOURCE_SYSTEM_MEMORY { + continue; + } + + if (res_desc.resource_attribute & hob::MEMORY_ATTRIBUTE_MASK) != hob::TESTED_MEMORY_ATTRIBUTES { + continue; + } + + let total_needed = MemoryBinManager::calculate_total_bin_size( + memory_type_info, + res_desc.physical_start + res_desc.resource_length, + ); + + if res_desc.resource_length >= total_needed { + result = Some((res_desc.physical_start, res_desc.resource_length)); + } + } + + // Reject if multiple Resource Descriptor HOBs with the owner GUID were found to avoid ambiguity + if count > 1 { + log::warn!( + target: LOG_TARGET, + "Multiple MemoryTypeInformation Resource Descriptor HOBs found ({}), rejecting all.", + count + ); + return None; + } + + if let Some((start, length)) = result { + log::info!( + target: LOG_TARGET, + "Found MemoryTypeInformation Resource Descriptor HOB: base={:#X} length={:#X}", + start, + length + ); + } else { + log::info!( + target: LOG_TARGET, + "No MemoryTypeInformation Resource Descriptor HOB found. DXE will allocate bins." + ); + } + + result +} + +/// Extracts the Memory Type Information from the GUID HOB. +/// +/// Returns a Vec of `EFiMemoryTypeInformation` entries with page counts aligned to +/// the appropriate granularity for each memory type. +pub(crate) fn extract_memory_type_info_from_hob(hob_list: &HobList) -> Option> { + hob_list.iter().find_map(|hob_entry| { + if let Hob::GuidHob(hob, data) = hob_entry { + if hob.name != MEMORY_TYPE_INFO_HOB_GUID.into_inner() { + return None; + } + + let entry_size = core::mem::size_of::(); + if data.is_empty() || data.len() > (EFI_MAX_MEMORY_TYPE + 1) * entry_size { + log::error!(target: LOG_TARGET, "Invalid Memory Type Information HOB data size: {}", data.len()); + return None; + } + + log::info!( + target: LOG_TARGET, + "Found Memory Type Information HOB ({} bytes, {} entries)", + data.len(), + data.len() / entry_size + ); + + let ptr = data.as_ptr() as *const EFiMemoryTypeInformation; + let len = data.len() / entry_size; + + // SAFETY: HOB data is 8-byte aligned per the PI spec. + // A compile-time assertion in allocator.rs verifies EFiMemoryTypeInformation's alignment requirement + // is <= 8 bytes. + let raw_entries = unsafe { core::slice::from_raw_parts(ptr, len) }; + + let mut entries: Vec = Vec::with_capacity(len); + for entry in raw_entries { + if entry.memory_type as usize >= EFI_MAX_MEMORY_TYPE { + // Either the sentinel or an invalid type. Include as-is (since the sentinel terminates processing). + entries.push(EFiMemoryTypeInformation { + memory_type: entry.memory_type, + number_of_pages: entry.number_of_pages, + }); + break; + } + + // Align page count to the type's allocation granularity for logging. + // The config table retains the original HOB values. Alignment is only applied when + // allocating the actual GCD bin region. + let granularity = MemoryBinManager::granularity_for_type(entry.memory_type); + let unaligned_size = uefi_pages_to_size!(entry.number_of_pages as usize); + let aligned_size = align_up(unaligned_size, granularity).unwrap_or(unaligned_size); + let aligned_pages = uefi_size_to_pages!(aligned_size); + + log::info!( + target: LOG_TARGET, + " MemTypeInfo: {} pages={} (GCD alloc will use {})", + memory_type_name(entry.memory_type), + entry.number_of_pages, + aligned_pages, + ); + + entries.push(*entry); + } + + Some(entries) + } else { + None + } + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use patina::base::{UEFI_PAGE_SIZE, align_pages_to_granularity}; + + const RT_GRAN_PAGES: u64 = + (MemoryBinManager::granularity_for_type(efi::RUNTIME_SERVICES_DATA) / UEFI_PAGE_SIZE) as u64; + + /// Returns the preferred allocation range for the given memory type. + /// + /// Returns `Some((base, max))` if a bin exists for this type with pages > 0. + fn preferred_range( + manager: &MemoryBinManager, + memory_type: efi::MemoryType, + ) -> Option<(efi::PhysicalAddress, efi::PhysicalAddress)> { + manager.active_bins().find(|(mt, _, _, _)| *mt == memory_type).map(|(_, base, max, _)| (base, max)) + } + + /// Returns a range size large enough to hold `pages` of a runtime type including alignment padding. + fn rt_range_size(pages: u32) -> u64 { + let granularity = MemoryBinManager::granularity_for_type(efi::RUNTIME_SERVICES_DATA); + // Enough for the pages plus one unit of granularity for alignment padding. + (pages as u64) * UEFI_PAGE_SIZE as u64 + granularity as u64 + } + + /// Initializes a `MemoryBinManager` from the given memory type info at the given base address. + /// + /// Uses `contiguous_alloc_size` to compute a range large enough for all bins. + #[coverage(off)] + fn init_bins(manager: &mut MemoryBinManager, base: u64, info: &[EFiMemoryTypeInformation]) { + let range_size = MemoryBinManager::contiguous_alloc_size(info).unwrap() as u64; + assert!(manager.initialize_from_range(base, range_size, info), "init_bins failed"); + } + + #[test] + fn test_memory_bin_new_uninitialized() { + let manager = MemoryBinManager::new(); + assert!(!manager.is_initialized()); + assert_eq!(preferred_range(&manager, efi::RUNTIME_SERVICES_DATA), None); + assert_eq!(manager.max_additional_descriptors(), 0); + } + + #[test] + fn test_memory_bin_calculate_total_size_no_alignment() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_CODE, number_of_pages: 10 }, + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: 20 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let size = MemoryBinManager::calculate_total_bin_size(&info, 0); + assert_eq!(size, (10 + 20) * UEFI_PAGE_SIZE as u64); + } + + #[test] + fn test_memory_bin_initialize_from_range() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_CODE, number_of_pages: 4 }, + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: 8 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let mut manager = MemoryBinManager::new(); + let range_size = rt_range_size(4) + rt_range_size(8); + let range_start = 0x1000_0000u64; + + let result = manager.initialize_from_range(range_start, range_size, &info); + assert!(result); + assert!(manager.is_initialized()); + + // Bins should have been set up + let rt_code_range = preferred_range(&manager, efi::RUNTIME_SERVICES_CODE); + assert!(rt_code_range.is_some()); + let (base, max) = rt_code_range.unwrap(); + assert!(base >= range_start); + assert!(max < range_start + range_size); + + let rt_data_range = preferred_range(&manager, efi::RUNTIME_SERVICES_DATA); + assert!(rt_data_range.is_some()); + + // Non-bin types should return None + assert_eq!(preferred_range(&manager, efi::BOOT_SERVICES_DATA), None); + } + + #[test] + fn test_memory_bin_initialize_from_range_too_small() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_CODE, number_of_pages: 100 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let mut manager = MemoryBinManager::new(); + // Range is too small for 100 pages + let result = manager.initialize_from_range(0x1000_0000, UEFI_PAGE_SIZE as u64, &info); + assert!(!result); + assert!(!manager.is_initialized()); + } + + #[test] + fn test_memory_bin_record_allocation_in_bin() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: 64 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let mut manager = MemoryBinManager::new(); + let range_start = 0x1000_0000u64; + let range_size = rt_range_size(64); + manager.initialize_from_range(range_start, range_size, &info); + + // Record in-bin allocation. The page count is aligned up to granularity. + manager.record_allocation(efi::RUNTIME_SERVICES_DATA, 4); + assert_eq!( + manager.statistics[efi::RUNTIME_SERVICES_DATA as usize].current_number_of_pages, + align_pages_to_granularity(4, MemoryBinManager::granularity_for_type(efi::RUNTIME_SERVICES_DATA)) + ); + + // Record another in-bin allocation + let prev = manager.statistics[efi::RUNTIME_SERVICES_DATA as usize].current_number_of_pages; + manager.record_allocation(efi::RUNTIME_SERVICES_DATA, 2); + assert_eq!( + manager.statistics[efi::RUNTIME_SERVICES_DATA as usize].current_number_of_pages, + prev + align_pages_to_granularity(2, MemoryBinManager::granularity_for_type(efi::RUNTIME_SERVICES_DATA)) + ); + } + + #[test] + fn test_memory_bin_record_free() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: 64 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let mut manager = MemoryBinManager::new(); + let range_start = 0x1000_0000u64; + let range_size = rt_range_size(64); + manager.initialize_from_range(range_start, range_size, &info); + + manager.record_allocation(efi::RUNTIME_SERVICES_DATA, RT_GRAN_PAGES); + assert_eq!(manager.statistics[efi::RUNTIME_SERVICES_DATA as usize].current_number_of_pages, RT_GRAN_PAGES); + + manager.record_free(efi::RUNTIME_SERVICES_DATA, RT_GRAN_PAGES); + assert_eq!(manager.statistics[efi::RUNTIME_SERVICES_DATA as usize].current_number_of_pages, 0); + + // Free more than allocated. It should stop at 0. + manager.record_allocation(efi::RUNTIME_SERVICES_DATA, RT_GRAN_PAGES); + manager.record_free(efi::RUNTIME_SERVICES_DATA, 100); + assert_eq!(manager.statistics[efi::RUNTIME_SERVICES_DATA as usize].current_number_of_pages, 0); + } + + #[test] + fn test_memory_bin_peak_tracking() { + let bin_pages: u32 = 8; + let alloc_pages = (bin_pages as u64).max(RT_GRAN_PAGES) + RT_GRAN_PAGES; + + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: bin_pages }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let mut manager = MemoryBinManager::new(); + let range_start = 0x1000_0000u64; + let range_size = rt_range_size(bin_pages).max(rt_range_size(alloc_pages as u32)); + manager.initialize_from_range(range_start, range_size, &info); + + // Allocate enough to exceed the original bin size + manager.record_allocation(efi::RUNTIME_SERVICES_DATA, alloc_pages); + + // Peak should be updated in memory_type_information + let expected = + align_pages_to_granularity(alloc_pages, MemoryBinManager::granularity_for_type(efi::RUNTIME_SERVICES_DATA)); + assert_eq!(manager.memory_type_information()[0].number_of_pages, expected as u32); + } + + #[test] + fn test_memory_bin_apply_descriptors_fully_within() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: 4 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let mut manager = MemoryBinManager::new(); + init_bins(&mut manager, 0x2000_0000, &info); + + let (bin_base, bin_max) = preferred_range(&manager, efi::RUNTIME_SERVICES_DATA).unwrap(); + let bin_pages = uefi_size_to_pages!((bin_max - bin_base + 1) as usize); + + let mut buffer = [efi::MemoryDescriptor { + r#type: efi::CONVENTIONAL_MEMORY, + physical_start: bin_base, + virtual_start: 0, + number_of_pages: bin_pages as u64, + attribute: efi::MEMORY_WB, + }; 10]; + + let count = manager.apply_bin_descriptors(&mut buffer, 1); + assert_eq!(count, 1); + assert_eq!(buffer[0].r#type, efi::RUNTIME_SERVICES_DATA); + assert_ne!(buffer[0].attribute & efi::MEMORY_RUNTIME, 0); + } + + #[test] + fn test_memory_bin_apply_descriptors_starts_before() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: 4 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let mut manager = MemoryBinManager::new(); + init_bins(&mut manager, 0x2000_0000, &info); + + let (bin_base, bin_max) = preferred_range(&manager, efi::RUNTIME_SERVICES_DATA).unwrap(); + let bin_size = bin_max - bin_base + 1; + + // Entry starts 1 page before bin + let entry_start = bin_base - UEFI_PAGE_SIZE as u64; + let entry_pages = uefi_size_to_pages!((bin_size + UEFI_PAGE_SIZE as u64) as usize); + + let mut buffer = [efi::MemoryDescriptor { + r#type: efi::CONVENTIONAL_MEMORY, + physical_start: entry_start, + virtual_start: 0, + number_of_pages: entry_pages as u64, + attribute: efi::MEMORY_WB, + }; 10]; + + let count = manager.apply_bin_descriptors(&mut buffer, 1); + assert!(count >= 2); + + // First entry should be the pre-bin conventional memory + assert_eq!(buffer[0].r#type, efi::CONVENTIONAL_MEMORY); + assert_eq!(buffer[0].physical_start, entry_start); + + // Second entry should be the bin type + assert_eq!(buffer[1].r#type, efi::RUNTIME_SERVICES_DATA); + assert_eq!(buffer[1].physical_start, bin_base); + } + + #[test] + fn test_memory_bin_apply_descriptors_ends_after() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: 4 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let mut manager = MemoryBinManager::new(); + init_bins(&mut manager, 0x2000_0000, &info); + + let (bin_base, bin_max) = preferred_range(&manager, efi::RUNTIME_SERVICES_DATA).unwrap(); + let bin_size = bin_max - bin_base + 1; + + // Entry starts at bin_base, ends 1 page after bin + let entry_pages = uefi_size_to_pages!((bin_size + UEFI_PAGE_SIZE as u64) as usize); + + let mut buffer = [efi::MemoryDescriptor { + r#type: efi::CONVENTIONAL_MEMORY, + physical_start: bin_base, + virtual_start: 0, + number_of_pages: entry_pages as u64, + attribute: efi::MEMORY_WB, + }; 10]; + + let count = manager.apply_bin_descriptors(&mut buffer, 1); + assert_eq!(count, 2); + + // First entry should be the bin type + assert_eq!(buffer[0].r#type, efi::RUNTIME_SERVICES_DATA); + assert_eq!(buffer[0].physical_start, bin_base); + + // Second entry should be conventional memory after bin + assert_eq!(buffer[1].r#type, efi::CONVENTIONAL_MEMORY); + assert_eq!(buffer[1].physical_start, bin_max + 1); + } + + #[test] + fn test_memory_bin_apply_descriptors_spans_bin() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: 4 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let mut manager = MemoryBinManager::new(); + init_bins(&mut manager, 0x2000_0000, &info); + + let (bin_base, bin_max) = preferred_range(&manager, efi::RUNTIME_SERVICES_DATA).unwrap(); + let bin_size = bin_max - bin_base + 1; + + // Entry spans before and after bin + let entry_start = bin_base - UEFI_PAGE_SIZE as u64; + let entry_pages = uefi_size_to_pages!((bin_size + 2 * UEFI_PAGE_SIZE as u64) as usize); + + let mut buffer = [efi::MemoryDescriptor { + r#type: efi::CONVENTIONAL_MEMORY, + physical_start: entry_start, + virtual_start: 0, + number_of_pages: entry_pages as u64, + attribute: efi::MEMORY_WB, + }; 10]; + + let count = manager.apply_bin_descriptors(&mut buffer, 1); + assert_eq!(count, 3); + + // Pre-bin conventional memory + assert_eq!(buffer[0].r#type, efi::CONVENTIONAL_MEMORY); + assert_eq!(buffer[0].physical_start, entry_start); + + // Bin region + assert_eq!(buffer[1].r#type, efi::RUNTIME_SERVICES_DATA); + assert_eq!(buffer[1].physical_start, bin_base); + + // Post-bin conventional memory + assert_eq!(buffer[2].r#type, efi::CONVENTIONAL_MEMORY); + assert_eq!(buffer[2].physical_start, bin_max + 1); + } + + #[test] + fn test_memory_bin_apply_descriptors_no_overlap() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: 4 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let mut manager = MemoryBinManager::new(); + init_bins(&mut manager, 0x2000_0000, &info); + + // Entry far away from bin + let mut buffer = [efi::MemoryDescriptor { + r#type: efi::CONVENTIONAL_MEMORY, + physical_start: 0x8000_0000, + virtual_start: 0, + number_of_pages: 4, + attribute: efi::MEMORY_WB, + }; 10]; + + let count = manager.apply_bin_descriptors(&mut buffer, 1); + assert_eq!(count, 1); + assert_eq!(buffer[0].r#type, efi::CONVENTIONAL_MEMORY); + } + + #[test] + fn test_memory_bin_apply_descriptors_runtime_attribute() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_CODE, number_of_pages: 4 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let mut manager = MemoryBinManager::new(); + init_bins(&mut manager, 0x2000_0000, &info); + + let (bin_base, bin_max) = preferred_range(&manager, efi::RUNTIME_SERVICES_CODE).unwrap(); + let bin_pages = uefi_size_to_pages!((bin_max - bin_base + 1) as usize); + + let mut buffer = [efi::MemoryDescriptor { + r#type: efi::CONVENTIONAL_MEMORY, + physical_start: bin_base, + virtual_start: 0, + number_of_pages: bin_pages as u64, + attribute: efi::MEMORY_WB, + }; 10]; + + let count = manager.apply_bin_descriptors(&mut buffer, 1); + assert_eq!(count, 1); + assert_eq!(buffer[0].r#type, efi::RUNTIME_SERVICES_CODE); + // Runtime services code is a runtime type + assert_ne!(buffer[0].attribute & efi::MEMORY_RUNTIME, 0); + } + + #[test] + fn test_memory_bin_max_additional_descriptors() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_CODE, number_of_pages: 4 }, + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: 8 }, + EFiMemoryTypeInformation { memory_type: efi::ACPI_MEMORY_NVS, number_of_pages: 2 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let mut manager = MemoryBinManager::new(); + init_bins(&mut manager, 0x2000_0000, &info); + + // RT Code (special+runtime), RT Data (special+runtime), ACPI NVS (special) = 3 (special) memory bins + assert_eq!(manager.max_additional_descriptors(), 3 * 2); + } + + #[test] + fn test_memory_bin_seed_statistics_from_hob() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: 64 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let mut manager = MemoryBinManager::new(); + let range_start = 0x1000_0000u64; + let range_size = rt_range_size(64); + manager.initialize_from_range(range_start, range_size, &info); + + manager.seed_statistics_from_hob(efi::RUNTIME_SERVICES_DATA, 3); + assert_eq!( + manager.statistics[efi::RUNTIME_SERVICES_DATA as usize].current_number_of_pages, + align_pages_to_granularity(3, MemoryBinManager::granularity_for_type(efi::RUNTIME_SERVICES_DATA)) + ); + } + + #[test] + fn test_memory_bin_no_bins_when_not_initialized() { + let manager = MemoryBinManager::new(); + + // All operations should be no-ops when not initialized + let mut buffer = [efi::MemoryDescriptor { + r#type: efi::CONVENTIONAL_MEMORY, + physical_start: 0x1000_0000, + virtual_start: 0, + number_of_pages: 4, + attribute: 0, + }; 5]; + + let count = manager.apply_bin_descriptors(&mut buffer, 1); + assert_eq!(count, 1); // Unchanged + assert_eq!(preferred_range(&manager, efi::RUNTIME_SERVICES_DATA), None); + assert_eq!(manager.max_additional_descriptors(), 0); + } + + #[test] + fn test_memory_bin_double_initialization_rejected() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: 4 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let mut manager = MemoryBinManager::new(); + let range_start = 0x1000_0000u64; + let range_size = rt_range_size(4); + + assert!(manager.initialize_from_range(range_start, range_size, &info)); + // Second initialization should be rejected + assert!(!manager.initialize_from_range(range_start + 0x100_0000, range_size, &info)); + } + + #[test] + fn test_merge_descriptors() { + let mut buffer = [ + efi::MemoryDescriptor { + r#type: efi::CONVENTIONAL_MEMORY, + physical_start: 0x1000, + virtual_start: 0, + number_of_pages: 1, + attribute: efi::MEMORY_WB, + }, + efi::MemoryDescriptor { + r#type: efi::CONVENTIONAL_MEMORY, + physical_start: 0x2000, + virtual_start: 0, + number_of_pages: 1, + attribute: efi::MEMORY_WB, + }, + efi::MemoryDescriptor { + r#type: efi::RUNTIME_SERVICES_DATA, + physical_start: 0x3000, + virtual_start: 0, + number_of_pages: 2, + attribute: efi::MEMORY_WB | efi::MEMORY_RUNTIME, + }, + efi::MemoryDescriptor { r#type: 0, physical_start: 0, virtual_start: 0, number_of_pages: 0, attribute: 0 }, + efi::MemoryDescriptor { r#type: 0, physical_start: 0, virtual_start: 0, number_of_pages: 0, attribute: 0 }, + ]; + + let count = MemoryBinManager::merge_descriptors(&mut buffer, 3); + assert_eq!(count, 2); + assert_eq!(buffer[0].r#type, efi::CONVENTIONAL_MEMORY); + assert_eq!(buffer[0].number_of_pages, 2); + assert_eq!(buffer[1].r#type, efi::RUNTIME_SERVICES_DATA); + } + + #[test] + fn test_merge_descriptors_single_entry() { + let mut buffer = [efi::MemoryDescriptor { + r#type: efi::CONVENTIONAL_MEMORY, + physical_start: 0x1000, + virtual_start: 0, + number_of_pages: 4, + attribute: efi::MEMORY_WB, + }]; + let count = MemoryBinManager::merge_descriptors(&mut buffer, 1); + assert_eq!(count, 1); + } + + #[test] + fn test_merge_descriptors_gap_prevents_merge() { + let mut buffer = [ + efi::MemoryDescriptor { + r#type: efi::CONVENTIONAL_MEMORY, + physical_start: 0x1000, + virtual_start: 0, + number_of_pages: 1, + attribute: efi::MEMORY_WB, + }, + // Gap: 0x2000 is the end of the first, but second starts at 0x4000 + efi::MemoryDescriptor { + r#type: efi::CONVENTIONAL_MEMORY, + physical_start: 0x4000, + virtual_start: 0, + number_of_pages: 1, + attribute: efi::MEMORY_WB, + }, + ]; + let count = MemoryBinManager::merge_descriptors(&mut buffer, 2); + assert_eq!(count, 2); + } + + #[test] + fn test_calculate_total_bin_size_with_alignment() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: 1 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + let entry_size = UEFI_PAGE_SIZE as u64; + + // With bin_top=0, no alignment padding + let size_no_align = MemoryBinManager::calculate_total_bin_size(&info, 0); + assert_eq!(size_no_align, entry_size); + + // With a non-zero bin_top that is already aligned to the type's granularity, no padding + let granularity = MemoryBinManager::granularity_for_type(efi::RUNTIME_SERVICES_DATA) as u64; + let aligned_top = 0x1000_0000u64; + assert_eq!(aligned_top % granularity, 0, "test precondition: top must be granularity-aligned"); + let size_aligned = MemoryBinManager::calculate_total_bin_size(&info, aligned_top); + if entry_size.is_multiple_of(granularity) { + assert_eq!(size_aligned, entry_size); + } else { + assert!(size_aligned >= entry_size); + } + + // With an unaligned bin_top, alignment padding is required. + let size_unaligned = MemoryBinManager::calculate_total_bin_size(&info, 0x1000_0001); + assert!(size_unaligned > entry_size); + } + + #[test] + fn test_active_bins_returns_only_configured_types() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_CODE, number_of_pages: 4 }, + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: 0 }, + EFiMemoryTypeInformation { memory_type: efi::ACPI_MEMORY_NVS, number_of_pages: 2 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let mut manager = MemoryBinManager::new(); + init_bins(&mut manager, 0x3000_0000, &info); + + let bins: Vec<_> = manager.active_bins().collect(); + // RTCode (4 pages) and ACPI NVS (2 pages). RTData has 0 pages so excluded. + assert_eq!(bins.len(), 2); + assert_eq!(bins[0].0, efi::RUNTIME_SERVICES_CODE); + assert_eq!(bins[1].0, efi::ACPI_MEMORY_NVS); + } + + #[test] + fn test_record_allocation_ignored_for_non_special_type() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::BOOT_SERVICES_DATA, number_of_pages: 4 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let mut manager = MemoryBinManager::new(); + init_bins(&mut manager, 0x1000_0000, &info); + + // BSData is not a special type, so record_allocation should be a no-op + manager.record_allocation(efi::BOOT_SERVICES_DATA, 2); + assert_eq!(manager.statistics[efi::BOOT_SERVICES_DATA as usize].current_number_of_pages, 0); + } + + #[test] + fn test_record_allocation_counts_outside_bin_range() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: 4 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let mut manager = MemoryBinManager::new(); + init_bins(&mut manager, 0x1000_0000, &info); + + // Address outside the bin range is counted so BDS can see overflow. + manager.record_allocation(efi::RUNTIME_SERVICES_DATA, 2); + assert_eq!( + manager.statistics[efi::RUNTIME_SERVICES_DATA as usize].current_number_of_pages, + align_pages_to_granularity(2, MemoryBinManager::granularity_for_type(efi::RUNTIME_SERVICES_DATA)) + ); + } + + #[test] + fn test_seed_statistics_always_counted_for_special_types() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: 16 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let mut manager = MemoryBinManager::new(); + let range_start = 0x1000_0000u64; + let range_size = rt_range_size(16); + manager.initialize_from_range(range_start, range_size, &info); + + // All memory bin-type HOB allocations are counted regardless of address + manager.seed_statistics_from_hob(efi::RUNTIME_SERVICES_DATA, 5); + assert_eq!( + manager.statistics[efi::RUNTIME_SERVICES_DATA as usize].current_number_of_pages, + align_pages_to_granularity(5, MemoryBinManager::granularity_for_type(efi::RUNTIME_SERVICES_DATA)) + ); + + // A non-memory bin type is not tracked + manager.seed_statistics_from_hob(efi::BOOT_SERVICES_DATA, 3); + assert_eq!(manager.statistics[efi::BOOT_SERVICES_DATA as usize].current_number_of_pages, 0); + } + + #[test] + fn test_memory_type_information_returned_after_init() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_CODE, number_of_pages: 4 }, + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: 8 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let mut manager = MemoryBinManager::new(); + init_bins(&mut manager, 0x1000_0000, &info); + + let mti = manager.memory_type_information(); + assert_eq!(mti.len(), 3); + assert_eq!(mti[0].memory_type, efi::RUNTIME_SERVICES_CODE); + assert_eq!(mti[0].number_of_pages, 4); + assert_eq!(mti[1].memory_type, efi::RUNTIME_SERVICES_DATA); + assert_eq!(mti[1].number_of_pages, 8); + } + + #[test] + fn test_apply_descriptors_skips_non_conventional() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: 4 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + + let mut manager = MemoryBinManager::new(); + init_bins(&mut manager, 0x2000_0000, &info); + + let (bin_base, bin_max) = preferred_range(&manager, efi::RUNTIME_SERVICES_DATA).unwrap(); + let bin_pages = uefi_size_to_pages!((bin_max - bin_base + 1) as usize); + + // Entry is BSCode within the bin range so it should not be converted + let mut buffer = [efi::MemoryDescriptor { + r#type: efi::BOOT_SERVICES_CODE, + physical_start: bin_base, + virtual_start: 0, + number_of_pages: bin_pages as u64, + attribute: efi::MEMORY_WB, + }; 10]; + + let count = manager.apply_bin_descriptors(&mut buffer, 1); + assert_eq!(count, 1); + assert_eq!(buffer[0].r#type, efi::BOOT_SERVICES_CODE); + } + + #[test] + fn test_memory_type_name_known_and_unknown() { + assert_eq!(memory_type_name(efi::RUNTIME_SERVICES_DATA), "RuntimeServicesData"); + assert_eq!(memory_type_name(efi::BOOT_SERVICES_CODE), "BootServicesCode"); + assert!(memory_type_name(0xFFFF).starts_with("Unknown")); + } + + #[test] + fn test_contiguous_alloc_size_single_entry() { + let rt_data_pages = 10; + + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: rt_data_pages }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + let size = MemoryBinManager::contiguous_alloc_size(&info).unwrap(); + let raw = rt_data_pages as usize * UEFI_PAGE_SIZE; + let granularity = MemoryBinManager::granularity_for_type(efi::RUNTIME_SERVICES_DATA); + // Must be at least raw + one granularity unit of padding, rounded up to granularity. + assert!(size >= raw + granularity); + assert_eq!(size % granularity, 0); + } + + #[test] + fn test_contiguous_alloc_size_multiple_entries() { + let rt_code_pages = 4; + let rt_data_pages = 8; + let acpi_reclaim_pages = 2; + let total_pages: usize = (rt_code_pages + rt_data_pages + acpi_reclaim_pages) as usize; + + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_CODE, number_of_pages: rt_code_pages }, + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: rt_data_pages }, + EFiMemoryTypeInformation { memory_type: efi::ACPI_RECLAIM_MEMORY, number_of_pages: acpi_reclaim_pages }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + let size = MemoryBinManager::contiguous_alloc_size(&info).unwrap(); + + let raw = total_pages * UEFI_PAGE_SIZE; + assert!(size >= raw, "size {:#X} must be >= raw {:#X}", size, raw); + } + + #[test] + fn test_contiguous_alloc_size_empty() { + // Sentinel only, no pages. + let info = [EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }]; + assert_eq!(MemoryBinManager::contiguous_alloc_size(&info), None); + } + + #[test] + fn test_contiguous_alloc_size_all_zero_pages() { + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: 0 }, + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_CODE, number_of_pages: 0 }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + assert_eq!(MemoryBinManager::contiguous_alloc_size(&info), None); + } + + #[test] + fn test_contiguous_alloc_size_skips_zero_page_entries() { + let rt_data_pages = 4; + + let info = [ + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_CODE, number_of_pages: 0 }, + EFiMemoryTypeInformation { memory_type: efi::RUNTIME_SERVICES_DATA, number_of_pages: rt_data_pages }, + EFiMemoryTypeInformation { memory_type: EFI_MAX_MEMORY_TYPE as u32, number_of_pages: 0 }, + ]; + let size = MemoryBinManager::contiguous_alloc_size(&info).unwrap(); + // Only 1 entry with pages > 0, so padding = 1 * max_granularity. + let raw = rt_data_pages as usize * UEFI_PAGE_SIZE; + assert!(size >= raw); + } +} diff --git a/sdk/patina/src/base.rs b/sdk/patina/src/base.rs index cc568aceb..7f5e4d0ef 100644 --- a/sdk/patina/src/base.rs +++ b/sdk/patina/src/base.rs @@ -322,6 +322,46 @@ where Ok((aligned_base, aligned_length)) } +/// Rounds a UEFI page count up to the nearest multiple of pages that correspond to the given +/// byte-level granularity. +/// +/// On architectures with a page allocation granularity larger than `UEFI_PAGE_SIZE` (e.g., +/// AARCH64 with 64KB runtime pages), the GCD allocates in granularity-sized chunks. This +/// function aligns a raw page count to match the actual GCD consumption. +/// +/// # Parameters +/// - `pages`: The raw page count to align. +/// - `granularity`: The allocation granularity in bytes (must be >= `UEFI_PAGE_SIZE`). +/// +/// # Returns +/// The page count rounded up to the nearest multiple of `granularity / UEFI_PAGE_SIZE`. +pub const fn align_pages_to_granularity(pages: u64, granularity: usize) -> u64 { + let granularity_pages: u64 = (granularity / UEFI_PAGE_SIZE) as u64; + if granularity_pages <= 1 { + return pages; + } + pages.div_ceil(granularity_pages) * granularity_pages +} + +/// Converts a page-aligned byte granularity to the corresponding bit shift value. +/// +/// Returns the number of trailing zero bits in `alignment`, which is the shift needed to +/// convert between addresses and page frame numbers at that granularity. +/// +/// # Parameters +/// - `alignment`: The alignment in bytes. Must be a power of two and >= `UEFI_PAGE_SIZE`. +/// +/// # Errors +/// Returns [`EfiError::InvalidParameter`] if `alignment` is not a power of two or is smaller +/// than `UEFI_PAGE_SIZE`. +pub const fn page_shift_from_alignment(alignment: usize) -> Result { + let shift = alignment.trailing_zeros() as usize; + if !alignment.is_power_of_two() || shift < UEFI_PAGE_SHIFT { + return Err(EfiError::InvalidParameter); + } + Ok(shift) +} + /// Generates a UEFI-style signature from between 1 to 8 bytes, packing them into a u16, u32 /// or u64 as appropriate for the parameters passed. /// @@ -439,6 +479,62 @@ mod tests { assert!(align_range(100u64, 100u64, 3u64).is_err()); // not power of two } + + #[test] + fn test_align_pages_to_granularity_equal_to_page_size() { + // granularity == UEFI_PAGE_SIZE => granularity_pages == 1, pages returned unchanged + assert_eq!(align_pages_to_granularity(0, UEFI_PAGE_SIZE), 0); + assert_eq!(align_pages_to_granularity(1, UEFI_PAGE_SIZE), 1); + assert_eq!(align_pages_to_granularity(7, UEFI_PAGE_SIZE), 7); + } + + #[test] + fn test_align_pages_to_granularity_smaller_than_page_size() { + // granularity < UEFI_PAGE_SIZE => granularity_pages == 0 <= 1, pages returned unchanged + assert_eq!(align_pages_to_granularity(5, UEFI_PAGE_SIZE / 2), 5); + } + + #[test] + fn test_align_pages_to_granularity_two_pages() { + assert_eq!(align_pages_to_granularity(0, 2 * UEFI_PAGE_SIZE), 0); + assert_eq!(align_pages_to_granularity(1, 2 * UEFI_PAGE_SIZE), 2); + assert_eq!(align_pages_to_granularity(2, 2 * UEFI_PAGE_SIZE), 2); + assert_eq!(align_pages_to_granularity(3, 2 * UEFI_PAGE_SIZE), 4); + assert_eq!(align_pages_to_granularity(4, 2 * UEFI_PAGE_SIZE), 4); + } + + #[test] + fn test_align_pages_to_granularity_sixteen_pages() { + assert_eq!(align_pages_to_granularity(0, SIZE_64KB), 0); + assert_eq!(align_pages_to_granularity(1, SIZE_64KB), 16); + assert_eq!(align_pages_to_granularity(15, SIZE_64KB), 16); + assert_eq!(align_pages_to_granularity(16, SIZE_64KB), 16); + assert_eq!(align_pages_to_granularity(17, SIZE_64KB), 32); + } + + #[test] + fn test_page_shift_from_alignment_valid() { + assert_eq!(page_shift_from_alignment(UEFI_PAGE_SIZE).unwrap(), 12); + assert_eq!(page_shift_from_alignment(UEFI_PAGE_SIZE * 2).unwrap(), 13); + assert_eq!(page_shift_from_alignment(SIZE_64KB).unwrap(), 16); + assert_eq!(page_shift_from_alignment(SIZE_2MB).unwrap(), 21); + } + + #[test] + fn test_page_shift_from_alignment_below_page_size() { + // 1KB is a power of two but smaller than UEFI_PAGE_SIZE + assert!(page_shift_from_alignment(SIZE_1KB).is_err()); + // 2KB is a power of two but smaller than UEFI_PAGE_SIZE + assert!(page_shift_from_alignment(SIZE_2KB).is_err()); + } + + #[test] + fn test_page_shift_from_alignment_not_power_of_two() { + assert!(page_shift_from_alignment(0).is_err()); + assert!(page_shift_from_alignment(3).is_err()); + assert!(page_shift_from_alignment(0x1001).is_err()); + } + #[test] fn test_signature() { const TEST0: u16 = signature!('A'); diff --git a/sdk/patina/src/efi_types.rs b/sdk/patina/src/efi_types.rs index a28475b98..4424a2a8f 100644 --- a/sdk/patina/src/efi_types.rs +++ b/sdk/patina/src/efi_types.rs @@ -13,6 +13,15 @@ use r_efi::efi; use crate::error::EfiError; +/// The number of standard UEFI memory types defined by the UEFI specification. +/// +/// This is the sentinel value used as the terminator in `EFI_MEMORY_TYPE_INFORMATION` arrays. +/// It currently equals one past the last valid `efi::MemoryType` constant (`efi::UNACCEPTED_MEMORY_TYPE`). +pub const EFI_MAX_MEMORY_TYPE: usize = efi::UNACCEPTED_MEMORY_TYPE as usize + 1; + +/// Sentinel value indicating a memory type with no `MemoryTypeInformation` entry. +pub const INVALID_INFORMATION_INDEX: usize = EFI_MAX_MEMORY_TYPE; + /// A wrapper for the EFI memory types. #[derive(Debug, Copy, Clone, PartialEq, Eq)] #[repr(u32)] diff --git a/sdk/patina/src/pi/hob.rs b/sdk/patina/src/pi/hob.rs index 0850dd406..6a50b50e1 100644 --- a/sdk/patina/src/pi/hob.rs +++ b/sdk/patina/src/pi/hob.rs @@ -990,7 +990,7 @@ pub const MEMORY_TYPE_INFO_HOB_GUID: crate::BinaryGuid = crate::BinaryGuid::from_string("4C19049F-4137-4DD3-9C10-8B97A83FFDFA"); /// Memory Type Information GUID Extension Hob structure definition. -#[derive(Debug)] +#[derive(Debug, Clone, Copy)] #[repr(C)] pub struct EFiMemoryTypeInformation { /// Type of memory being described.