[ipxe-devel] Using iPXE, e820 RAM map badly changed, kernel panic over seeing only 50MB

Mon Aug 8 05:44:33 UTC 2016

Hi james,

I will only try to explain a few things that I know about.

On Mon, Aug 8, 2016 at 5:37 AM, james harvey <jamespharvey20 at gmail.com> wrote:
> Based on that git comment about x86_64 pcbios being non-functional, I
> set out to try making en EFI version.  I'll admit, I have never used
> EFI before, being content with the way things used to work and having
> nothing pushing me to it.
>
> So, instead of running:
>
>    make bin/15b3673c.rom
>
> I ran:
>
>    make bin-x86_64-efi/15b3673c.efirom
>
> Successfully burned it to my Mellanox MT26428 ConnectX, and
> successfully read its ROM it to a file and compared it to make sure it
> was burned correctly.
>
> On reboot, I no longer see the screen saying to hit CTRL+B to enter
> IPXE.  (Maybe that's supposed to go away when using EFI?)
>
> If I enter my BIOS's boot menu, it no longer shows the card with iPXE.
>
> If I enter my BIOS's EFI menu, running devices or drivers doesn't show
> the card with iPXE.
>
> I see http://ipxe.org/download says some network cards can't support
> ROMs larger than 64kB.  That might only refer to pcbios ROMS, but
> anyways both are over 64kB.  The iPXE pcbios ROM is 77,312 bytes, and
> the iPXE efi ROM is 155,648 bytes.  So, I don't think that's the
> issue.

Indeed this is probably not related, the >64kB is normal with iPXE and
to reach <=64kB (or in some cases even <=56kB) a lot of default
features needs to be disabled, the virtualbox config is constrained to
the 56kB limit. (but might not reach it in ipxes current form)

To get EFI PXE enabled you need to enable the "network stack" in most
bioses. Do note tho that at this time ipxe EFI does not support iSCSI
(which might or might not be relevant to your usecase)

>
> On Sun, Aug 7, 2016 at 10:27 PM, james harvey <jamespharvey20 at gmail.com> wrote:
>> I can boot an unmodified Arch Linux ISO from a DVD, and kernel
>> correctly sees 64GB, reporting:
>>
>> [    0.000000] e820: BIOS-provided physical RAM map:
>> [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009e7ff] usable
>> [    0.000000] BIOS-e820: [mem 0x000000000009e800-0x000000000009ffff] reserved
>> [    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
>> [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000392f6fff] usable
>> [    0.000000] BIOS-e820: [mem 0x00000000392f7000-0x0000000039e69fff] reserved
>> [    0.000000] BIOS-e820: [mem 0x0000000039e6a000-0x000000003a109fff] usable
>> [    0.000000] BIOS-e820: [mem 0x000000003a10a000-0x000000003acb8fff] ACPI NVS
>> [    0.000000] BIOS-e820: [mem 0x000000003acb9000-0x000000003b613fff] reserved
>> [    0.000000] BIOS-e820: [mem 0x000000003b614000-0x000000003b614fff] usable
>> [    0.000000] BIOS-e820: [mem 0x000000003b615000-0x000000003b69afff] reserved
>> [    0.000000] BIOS-e820: [mem 0x000000003b69b000-0x000000003b8b7fff] usable
>> [    0.000000] BIOS-e820: [mem 0x000000003b8b8000-0x000000003bff8fff] reserved
>> [    0.000000] BIOS-e820: [mem 0x000000003bff9000-0x000000003bffffff] usable
>> [    0.000000] BIOS-e820: [mem 0x000000003c000000-0x000000003dffffff] reserved
>> [    0.000000] BIOS-e820: [mem 0x0000000040000000-0x000000004fffffff] reserved
>> [    0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed24fff] reserved
>> [    0.000000] BIOS-e820: [mem 0x00000000fed30000-0x00000000fed34fff] reserved
>> [    0.000000] BIOS-e820: [mem 0x00000000fed40000-0x00000000fed44fff] reserved
>> [    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
>> [    0.000000] BIOS-e820: [mem 0x0000000100000000-0x00000010bfffffff] usable
>> [    0.000000] debug: ignoring loglevel setting.
>>
>> If I boot the SAME unmodified Arch linux ISO through iPXE (from
>> another machine sharing it using SRP), kernel only sees about 50MB,
>> reporting:
>>
>> [    0.000000] e820: BIOS-provided physical RAM map:
>> [    0.000000] BIOS-88: [mem 0x0000000000000000-0x000000000009efff] usable
>> [    0.000000] BIOS-88: [mem 0x0000000000100000-0x00000000032bbfff] usable
>> [    0.000000] debug: ignoring loglevel setting.
>>
>> I believe iPXE is modifying the e820 information the linux kernel
>> retrieves.  Both because almost no memory is seen, and the kernel
>> gives each block of memory referring to "BIOS-88" rather than
>> "BIOS-e820".
>>
>> I saw in the iPXE code fakee820.{h,c} and e820mangler.S.  In
>> e820mangler.S, I see references to "morons at ACPI" causing problems
>> in this area before, so I think I'm on the right track on what's going
>> on, rather than it being a kernel bug, since at this point the kernel
>> shouldn't be behaving any differently, and until 2 more kernel seconds
>> doesn't try to do anything differently having been loaded by SRP.
>>
>> I am developing an ib_sbft kernel module and a program to use it, to
>> be able to complete SRP booting using your SBFT ACPI table, and one of
>> the most active linux Infiniband kernel developers has agreed to look
>> over, help test, and help get accepted, based on how iscsi works with
>> the kernel module iscsi_ibft.  I actually think the kernel module is
>> about done, as it correctly parses randomly made SBFT tables using
>> your structure on an already booted system - I'm now trying to test it
>> using your SBFT tables.
>>
>> But I'm getting a kernel panic that the 27MB initrd is too large due
>> to not having enough memory, which led me here.
>>
>> Looking at the git log for the e820 source files, I saw a very
>> confusing (to me) git comment:
>>
>>     [bios] Add bin-x86_64-pcbios build platform
>>
>>     Move most arch/i386 files to arch/x86, and adjust the contents of the
>>     Makefiles and the include/bits/*.h headers to reflect the new
>>     locations.
>>
>>     This patch makes no substantive code changes, as can be seen using a
>>     rename-aware diff (e.g. "git show -M5").
>>
>>     This patch does not make the pcbios platform functional for x86_64; it
>>     merely allows it to compile without errors.
>>
>> So I'm confused by "does not make the pcbios platform functional for
>> x86_64".  I am on a 64-bit system, a MSI MS-7885/X99S SLI Plus
>> motherboard, an Intel i7 5820K, and 64GB memory.
>>

The "not make the pcbios platform functional" refers to iPXE at the
time not being possible to compile or used in legacy x86_64 mode,
which was needed to support PXE BAR ROMs located above 4GB. This has
since changed (take a look on the commits right after the above one)

make bin-x86_64-pcbios/15b3673c.rom

should get you a normal legacy rom, but that only runs on 64bit machines

>>
>>
>> Also below is the kernel panic through iPXE.  Using an earlier made
>> ISO that had a 24MB initrd gets 2 kernel seconds past this point, to
>> where the kernel attempts mounting the new root, which is where my
>> kernel module + other program will come in.
>>
>> ....

/Christian