[ipxe-devel] SRP booting linux, SRP volume with ISO stops seeing itself

[james harvey]

Before I keep going down the hole, let me make sure my plan isn't
futile.  I want to make a diskless system, by having iPXE in my
InfiniBand card's ROM, and being able to boot an Arch Linux Live ISO
(custom built with srp kernel modules) exported as a SRP volume backed
by the ISO, and install on other SRP volumes which are backed by LVM
volumes.

I have a real general idea of how the iPXE to kernel handoff works,
through the iBFT (iSCSI Boot Firmware Table.)  Since I'm using SRP
rather than iSCSI, is the iBFT still used?  If not, how does the SRP
connection get handed off, with all the variables needed for an SRP
connection?

My preference is to just use iPXE scripting and sanboot, and not use
pxelinux.  This is fine for this situation, right?  I just want to be
able to boot Arch Linux.  (Any Windows I run will be inside KVM.)

Here's where I'm at now, having burned iPXE's 15b3673c.rom to my
Mellanox card's ROM using mstflint.

======= Try booting lun 3, a fileio backstore srp target of
archlinux-2016.01.01-dual.iso =======
iPXE> dhcp
Waiting for link-up on net0.............. ok
Configuring (net 0 00:02:c9:4b:52:75)...... ok
iPXE> sanhook --drive 0x80
ib_srp::1d534b0003c90200::fe800000000000000002c903004b531d:ffff:0002c903004b531c:4:0002c903004b531c:0002c903004b531c
{{{this line in red}}} IBSRP 0x26764 for
fe800000:00000000:0002c903:004b531d 0002c903:004b531c
Registered SAN device 0x80
iPXE> sanboot
============

I finally get to the ISO boot menu!  I select boot x86_64.  Right
after "triggering uevents", the screen blanks, goes into a higher
resolution, and I see 13 garbage characters (hi ascii, many spaces,
and they're colored pink) followed by "dd=off to disable)... ok".  It
gets to "Mounting '/dev/disk/by-label/ARCH_201601' to
'rom/archiso/bootmnt'.  Then it says the dreadful "Waiting 30 seconds
for device /dev/disk/by-label/ARCH_201601 ..." which times out and
dumps me to an early shell.  Running blkid shows my locally installed
hard drives (which I'll be removing once I have this working), but
doesn't show the SRP-mounted cdrom drive.

The infiniband kernel module ib_srp and any dependency ib_* modules
aren't on the ISO.  (In Arch, mkinitcpio is used, and I checked the
ISO's archiso.img which is the ISO's initramfs, and no ib_* kernel
modules are present.)  This is what's causing the ISO to suddenly not
see itself, right?

I'm also thinking there's nothing giving the initrd /
initramfs-linux.img all of the SRP variables.  How in the world is
that handled?

I'm also thinking srptools isn't on the ISO.  Does it need to be, or
is there a way around that, like kernel options?  But, even then, I'm
not sure how to hand off any information like that from within an iPXE
script calling sanboot...

Any pointers would be SO MUCH appreciated!