[ipxe-devel] iPXE memory usage

[Haggai Eran]

Hi,

I've been trying to get a picture of how iPXE uses memory when run as an
option ROM. I was hoping I could get feedback from the list, whether I'm
in the right direction, things I've missed, and also ask a couple of
questions.

I am focusing on iPXE running as an option ROM with the mrom prefix, and
assuming a PCI 3 compatible BIOS, with PMM and PnP working. I'm also
assuming the user hasn't asked for a shell during POST. The numbers are
from a build I made, but I'm sure they can vary.

Initialization
**************
During POST, iPXE will call PMM twice to get two temporary extended
memory regions:
* Image source - to contain the compressed binary (about 80k).
* Decompression area - to contain the decompressed binary temporarily
(about 640k).
The initialization also copies the prefix to the option ROM segment
provided by the BIOS, which takes about 3k.

Boot entry vector
*****************
The boot entry vector does the following memory allocations (and related
initialization work):
* Allocates about 13k for 16-bit code and data segments in conventional
memory, by modifying the memory size in the BDA (address 0x413), and
copies compressed data to this area.
* Copy the compressed image from the PCI device's expansion ROM BAR to
the image source section (allocated by the PMM).
* Decompress from the image source section to the decompression area.
* Relocate the 32-bit code and data section from the decompression area
to the highest memory area available according to the BIOS's e820 memory
map.

C code
******
* Hook int 15h to hide used memory regions from the e820 memory map.
This covers the 16-bit sections in conventional memory, 32-bit ipxe code
and data at the top of memory, and umalloc allocations.
* In C code, umalloc allocations will be served from the top of
available memory, just below where iPXE is located.

When are the regions above freed?
*********************************
* PMM allocations are implicitly freed when POST ends.
* int 15h is unhooked on PXENV_STOP_UNDI, or when boot fails. It is
never released on SAN boot (e.g. with iSCSI).
* Conventional memory is only freed when boot fails.
* Option ROM area is never freed.

Questions
*********
* Why is relocation needed? Is it to allow the umalloc mechanism to
allocate memory near the where iPXE was relocated? Or to avoid having
holes in the middle of the memory map, where the PMM might have decided
to allocate the memory?

* According to the PCI 3 firmware specs temporary memory allocated by
the PMM is usable only within the expansion ROM's initialization.
However, if I understand correctly, iPXE will use PMM regions even after
POST ends. Have you encountered BIOS implementations that enforce this
requirement? Wouldn't it be better to use the e820 map to allocate the
image source area and the decompression area when running after POST?

* The PCI 3 specs also recommend using the PMM for conventional memory
allocations. It also specifies a rather complicated method for using
EBDA in the option ROM, which includes moving the EBDA down in memory
before usage, and always using indirect pointers to it. I'm wondering if
the current method used with iPXE is safe if another option ROM uses the
method in the spec, and whether you know of option ROMs that do use that
method.

Thanks for your help,
Haggai