[ipxe-devel] iPXE, ESXi 5.5 Stateless + Caching Install - BMP Razor + Chef Integration, Routed iSCSI, IaaS block

Wed Jul 23 17:52:17 UTC 2014

Hello Andrew,

Thanks for the reply...

Re: iSRB - Initial SAN Remote Boot volume.  I must of made it up a couple of months back :-)

Storage services (iSCSI/NFS) are one hop away on VLAN 50 and 44.  I'm trying to attach a iSCSI LUN on 10.50.0.100.

Currently PXE booting via the Intel X540 NIC, which is connected to a 10G trunk port (x2) with a native VLAN of 2003. 

A DHCP relay between VLAN 2003 and 10.136.0.10 (VLAN 136) is in place works.  The DHCP service on VLAN 136 (10.136.0.10) gives out a static address to the blade (10.200.3.13).   

Relative DHCP server config;

subnet 10.200.3.0 netmask 255.255.255.0 {
  range 10.200.3.10 10.200.3.100;
        option routers                  10.200.3.4;
        option subnet-mask              255.255.255.0;
        option broadcast-address        10.200.3.255;
        option domain-name-servers      10.121.0.10;
        option ntp-servers              10.121.0.10;
        option rfc3442-classless-static-routes
                24, 10, 50, 0,          10, 200, 3, 1,
                24, 10, 44, 0,          10, 200, 3, 1;

host BM1-QVSL-ESXI-03-003 {
	hardware ethernet a0:36:9f:0e:7b:58;
        fixed-address 10.200.3.13;
}
  if exists user-class and option user-class = "iPXE" {
    filename "bootstrap.ipxe";
  } else {
    filename "undionly.kpxe";
  }
next-server 10.136.0.10;
}

...for basic testing, in the bootstrap.ipxe file i have;

#!ipxe

# The number of tries made to obtain DHCP configuration to date, and the
# maximum number to make before we give up.  Each time DHCP failed we will
# sleep one secord per attempt already made, starting at one second, and
# ramping up as we go.
set tries:uint32    0
set maxtries:uint32 60

:retry

isset ${ip} || goto dhcp_net0
set dhcp_mac ${mac:hexhyp}

goto attach_solidfire

:dhcp_net0
isset ${net0/mac} && dhcp net0 || goto dhcp_net1
echo net0 has DHCP
set dhcp_mac net0$${net0/mac:hexhyp}

:dhcp_net1
isset ${net1/mac} && dhcp net1 || goto dhcp_net2
echo net1 has DHCP
set dhcp_mac net1$${net1/mac:hexhyp}

:dhcp_net2
isset ${net2/mac} && dhcp net2 || goto dhcp_net3
echo net2 has DHCP
set dhcp_mac net2$${net2/mac:hexhyp}

:dhcp_net3
isset ${net3/mac} && dhcp net3 || goto dhcp_net4
echo net3 has DHCP
set dhcp_mac net3$${net3/mac:hexhyp}

:dhcp_net4
isset ${net4/mac} && dhcp net4 || goto chain_boot
echo net4 has DHCP
set dhcp_mac net4$${net4/mac:hexhyp}

goto attach_solidfire

:attach_solidfire
echo Attaching iSCSI Boot Volume 645
set username "QVSL-ZONE-3"
set password "x/z'Gt5bB28n~j:7"
set root-path iscsi:10.50.0.100::::iqn.2010-01.com.solidfire:jqqi.qvsl-boot-z3b3.645
sanhook --drive 0x80 ${root-path} || goto failed

goto chain_boot

:chain_boot
chain http://10.136.0.10:8080/svc/boot?net0=${net0/mac:hexhyp}&net1=${net1/mac:hexhyp}&net2=${net2/mac:hexhyp}&net3=${net3/mac:hexhyp}&net4=${net4/mac:hexhyp}&dhcp_mac=
${dhcp_mac}&serial=${serial}&asset=${asset}&uuid=${uuid} || goto error
echo successfully completed the chain-loaded script; booting locally with 'exit'
exit

:error
iseq ${tries} ${maxtries} && goto failed
inc tries
sleep ${tries}
goto retry

:failed
echo failed to obtain DHCP data after ${tries} attempts, giving up.
sleep 60
reboot

---

The sanhook command fails.  Happy to share a diagram, its non-production - nothing special.  I need iPXE to install a route for 10.50.0.0/24 and 10.44.0.0/24 via 10.200.3.1.  Was hoping to do that via the option rfc3442 section in ISC DHCP.

        option rfc3442-classless-static-routes
                24, 10, 50, 0,          10, 200, 3, 1,
                24, 10, 44, 0,          10, 200, 3, 1;

When I break the iPXE boot after it gets an address and type route, is don't see the additional routes.  Given this, I suspect the routes received via the DHCP server are not being installed.  Tried manually set the routes for 10.50.0.0/24 in the iPXE shell, but I can't figure out the syntax to use. 

Thanks for your help! :)

Mike

________________________________________
From: Andrew Bobulsky [rulerof at gmail.com]
Sent: Tuesday, July 22, 2014 2:08 PM
To: Mike Harris
Cc: ipxe-devel at lists.ipxe.org
Subject: Re: [ipxe-devel] iPXE, ESXi 5.5 Stateless + Caching Install - BMP Razor + Chef Integration, Routed iSCSI, IaaS block

Hello Mike,

On Tue, Jul 22, 2014 at 9:44 AM, Mike Harris <mharris at quovadis.bm> wrote:
> Greetings!
>
> I'm currently using local storage (2-way mirror, LSI controller) to booting
> a "test rabbit" SuperMicro blade in my lab.
>
> High level;
>
> + X8DTT-H Motherboard
> + Intel X540 NIC (dual 10G copper)
>
> We commonly use these blades and would like to bare metal provision a failed
> blade from a last known good state.  Final profile applied by an ESM tool
> like Chef.
>
> Currently the blade use DAS.  It works, but an iSRB is better and more
> convenient (from a hardware point of view).
>
> I would like to;
>
> + iPXE boot,
> + Attach an SAN volume for a boot device (1),
> + have Razor; factor, tag, kickstart the ESXi 5.5 install process.
> + Then broker the node to Chef for final provisioning.

Everything you've laid out here sounds pretty normal.  Though I can't
claim to know what "iSRB" is, and Google wasn't much help either ;)

> I've been able to get most of this working aside from the SAN volume (1).
> Routed iSCSI/NFS SAN volume is a challenge since the default iPXE binary
> doesn't support vcreate.

Are you working with a burned-in iPXE ROM, then?  Even with iPXE
built-in to your systems, you can still use that ROM to chainload a
more feature-complete iPXE binary from one of your servers.  Have a
look at the links here for an idea of how to do it with dhcpd:
http://ipxe.org/examples#using_chainloading_to_improve_the_feature_set_of_ipxe

Alternatively, you might even be able to do that with an iPXE script
that chainloads a different iPXE if the vcreate command fails.  Keep
in mind that this is just a guess, but I'd certainly try it if
adjusting my DHCP server was off the table.  Something like:

vcreate --tag 100 && goto attachMySAN || goto ipxeUpgrade

Some cursory testing shows me that it'd work: http://i.imgur.com/S0yPnix.png

But in the interest of Doing it Right[TM], something like this might
be most appropriate: http://pastebin.com/raw.php?i=bDV5M1X6

> I haven't found any exampled of routed iSCSI (or
> NFS), I'm sure someone has, hopefully they're on this mailing list.

Routed iSCSI/NFS?  Unless you mean something different... iPXE's iSCSI
and NFS clients shouldn't have a problem talking to a server in a
different subnet, assuming that there's no firewall/NAT/ALG/whatever
in the way to mess with your traffic.

> If anyone has any tips on routed iSCSI/NFS boot volumes, and Razor/Chef
> integration experience, I'd be move appreciative for some feedback on how
> you managed iSRB.  I have a pretty network diagram of the POC which I'm
> happy to share if you're interested.

Diagrams couldn't hurt, though if you're looking to limit the scope of
who sees them, chances are pretty high that exposing such things to my
eyeballs won't yield a significant return.  Even in spite of my
enthusiasm for iPXE sorcery ;)

> Although the reward is strictly karma at this point, I may have a bunch of
> Chef work that needs doing that could lead to some meaningful PS for a
> couple of ninjas in a cool location or two.
>
> May the force be with you!
>
> Mike

The force in strong in this list! ;)

Best Regards,
Andrew Bobulsky