[ipxe-devel] Best practices for troubleshooting UEFI General Protection Exceptions when booting iPXE?

[Capriccio, Peter]

I was recently working on a ProLiant DL380 Gen9 system (which, unfortunately, I no longer have access to) where booting an iPXE "snponly.efi" image from a certain interface would fail with a General Protection Exception.

This system had the onboard 1Gb Ethernet, 4-port "331i" adapter (Broadcom-based), plus an add-on InfiniBand QDR/Ethernet 10Gb 2-port "544+FLR-QSFP" adapter (Mellanox ConnectX-3 Pro-based) running in 10Gb Ethernet mode. The system was running in its UEFI mode (it also supports running in a Legacy BIOS mode). I was able to boot iPXE successfully from the 331i adapter but not from the 544+FLR-QSFP adapter -- the latter throwing the exception. However, I was able to boot to elilo 3.16 and SYSLINUX 6.03 (EFI) from the 544+FLR-QSFP adapter.

I was using iPXE build "d0bfd83" from:

  https://git.ipxe.org/ipxe.git/snapshot/d0bfd830e4e5ddd1015dda66833a99b068b6a519.tar.gz

- made no config file changes

- built using:

  # make bin-x86_64-efi/snponly.efi EMBED=/root/foo.ipxe

- where "/root/foo.ipxe" contained only:

  #!ipxe
  dhcp
  shell

Here is what the console output looked like (with a lot of blank lines removed) when booting from the 544+FLR-QSFP adapter:

>> Booting Embedded FlexibleLOM 1 Port 1 : HP InfiniBand QDR/Ethernet 10Gb 2-port 544+FLR-QSFP Adapter - NIC (PXE IPv4)

>> Booting PXE over IPv4.
  Station IP address is 10.7.161.131

  Server IP address is 10.7.170.254
  NBP filename is snponly.efi
  NBP filesize is 147616 Bytes
Downloading NBP file...

  NBP file downloaded successfully.
iPXE initialising devices...


X64 Exception Type 0D - General Protection Exception

RCX=000000007744D978 DX=0000000070DFE000 R8=0000000000500000 R9=0000000000000000
RSP=000000006BA23108 BP=000000006BA2D971 AX=0000000000000038 BX=000000007744D978
R10=0000000000000000 11=000000006BA23130 12=000000000000001A 13=000000006BA233B0
R14=0000000000000000 15=0000000000000006 SI=0000000000000001 DI=000000006BA232D0
CR2=0000000000000000 CR3=000000006B983000 CR0=80000013 CR4=00000668 CR8=00000000
CS=0038 DS=0038 SS=0030 ES=0038 RFLAGS=00010203 MSR1D9=4801 345=32C4 1C9=000F

LBRs From      To       From      To       From      To       From      To
01h  00093009->74407191 78464454->0005E11A 7846443A->78464454 78466365->78464432
05h  74407BEB->78466340 0005E11A->74407BE0 778F2F27->00000000 778F1F89->778F2ED7
09h  778F0B34->778F1EEF 778F0A94->778F0AAF 778F0A61->778F0A7C 778F0A37->778F0A52
0Dh  778F1B08->778F09FA 778F1ACC->778F1AF5 778F1AA8->778F1AC0 7440719C->78466340
Null Caller: ImageBase=00000000778EB000 CALL ImageBase        ImageName+Offset
00h  0000000000000000 No Image Information


CALL ImageBase        ImageName+Offset


STACK   00h      04h      08h      0Ch      10h      14h      18h      1Ch
RSP+00h 778F2F2A 00000004 6BA3A81C 000025EE 779059D1 778F1F8B 712FDFFF 72372398
RSP+20h 00000001 72371018 72372398 778F0B36 00000008 72371018 6BA23306 778F1B0B
RSP+40h 71487068 6BA2B776 71487018 6BA232B0 778F18FA 70DFD000 6BA232B0 6BA232B0
RSP+60h 77445439 00000000 6BA2B7D4 6BA23306 C0000000 7744D978 7744554C 00009CC0
RSP+80h 000070DF 00000000 7744D978 77445D88 7744D788 72372398 00000002 72372398
RSP+A0h 6BA233B0 71487538 72F72CA8 774453AD 00000000 00400100 7744D978 72372398
RSP+C0h 00000002 7744D688 7744C3D8 7744C438 77441C64 72372398 7744D6D8 00000002
RSP+E0h 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Just something to note (not sure if it matters), the 544+FLR-QSFP adapter has its own iPXE stack in its firmware, although I only see output from it when the system is in Legacy BIOS mode.

Now, if you've made it this far :), my questions are these:

- What is the best method, in general terms, for troubleshooting a problem such as this?

- What debug flags (or breakpoints maybe?) would produce the most fruitful data?

- What other information needs to be gathered?

Sorry for the noob questions, but as a non-developer, there is not a lot for me to go on when running into these kinds of problems, so the next time I come across this, I want to be better prepared.

Thanks for any feedback,

Pete C. (employed by, but not speaking for, HPE Financial Services)