[ipxe-devel] SRP booting linux, SRP volume with ISO stops seeing itself

[Shao Miller]

On 1/7/2016 21:19, james harvey wrote:
> Before I keep going down the hole, let me make sure my plan isn't
> futile.  I want to make a diskless system, by having iPXE in my
> InfiniBand card's ROM, and being able to boot an Arch Linux Live ISO
> (custom built with srp kernel modules) exported as a SRP volume backed
> by the ISO, and install on other SRP volumes which are backed by LVM
> volumes.
>

"Custom-built" is a key point.  That means you can have whatever you 
want inside an initramfs that'll be in that ISO9660 FS image.

Having typed that, I'd ask: Why use an .ISO file, at all?  iPXE can boot 
your kernel and initramfs via HTTP, for instance.

Are you using PXE parameters and/or scripts?  Are any of those 
parameters and/or any of the scripts booted-system-specific, or does 
every booted system get the same stuff?

> [...]
>
> I finally get to the ISO boot menu!  I select boot x86_64.  Right
> after "triggering uevents", the screen blanks, goes into a higher
> resolution, and I see 13 garbage characters (hi ascii, many spaces,
> and they're colored pink) followed by "dd=off to disable)... ok".  It
> gets to "Mounting '/dev/disk/by-label/ARCH_201601' to
> 'rom/archiso/bootmnt'.  Then it says the dreadful "Waiting 30 seconds
> for device /dev/disk/by-label/ARCH_201601 ..." which times out and
> dumps me to an early shell.  Running blkid shows my locally installed
> hard drives (which I'll be removing once I have this working), but
> doesn't show the SRP-mounted cdrom drive.
>

Once the Linux operating system's kernel takes control of the booted 
system, BIOS INTerrupt 0x13 is no longer used for disk/disc-access.  A 
nice flow might be:
- iPXE
- Linux kernel
- Custom initramfs
- System-specific delta of overrides added to the initramfs
- Initramfs connects to the data-store; establishes block devices
- Initramfs mounts to-be-root filesystem
- Pivot-root into the new root filesystem and continue with its initial 
process

> The infiniband kernel module ib_srp and any dependency ib_* modules
> aren't on the ISO.  (In Arch, mkinitcpio is used, and I checked the
> ISO's archiso.img which is the ISO's initramfs, and no ib_* kernel
> modules are present.)  This is what's causing the ISO to suddenly not
> see itself, right?
>

Not quite.  The .ISO doesn't have a perspective.  It [probably] has 
ISOLINUX, the boot-loader.  That boot-loader uses INT 0x13 to access the 
virtual disc (the .ISO).  Then the kernel is booted.

 From the kernel's perspective, there never was an .ISO, as life has 
just begun.  But yes, you'd need your drivers to be loaded in order to 
access the block device.

> I'm also thinking there's nothing giving the initrd /
> initramfs-linux.img all of the SRP variables.  How in the world is
> that handled?
>

I don't know if there is an iBFT counterpart for your scenario, but 
someone else might.

> I'm also thinking srptools isn't on the ISO.  Does it need to be, or
> is there a way around that, like kernel options?  But, even then, I'm
> not sure how to hand off any information like that from within an iPXE
> script calling sanboot...
>
> Any pointers would be SO MUCH appreciated!

If you can dump the drivers (kernel modules) and tools into the 
initramfs, boot it, then manually establish a connection to the block 
devices while in an emergency shell or some such, that would be a good 
proof-of-concept.

Does Arch Linux already have instructions for this kind of scenario?

--
Shao Miller
C. E. O.
Synthetel Corporation
T: +1.9053927729
E: sha0.miller at synthetel.com
W: https://www.synthetel.com