[ipxe-devel] iPXE boot fails if multique is enabled in Openstack

Wed Nov 22 15:57:35 UTC 2017

Hi Zoltan,

I'll work on setting this up on my host in the upcoming days. Adding
Victor to cc in case this is a known issue or he knows off-hand why
this doesn't work.

In summary, this is vhost-user networking configured with mq=on but
the simple iPXE driver uses only the first queue. Maybe incoming DHCP
packets are delivered to one of the other queues, which is why the
driver doesn't see them.

Thanks,
Ladi

On Mon, Nov 20, 2017 at 4:44 PM, Zoltan Kanizsai
<zoltan.kanizsai at ericsson.com> wrote:
> Hi Ladi,
>
> I'm in the same team with Rafael. Here are the information you asked for:
>
> Qemu related package versions:
>
> root at blade0-7:~# apt list|grep qemu
>
> WARNING: apt does not have a stable CLI interface yet. Use with caution in
> scripts.
>
> ipxe-qemu/trusty-updates,now 1.0.0+git-20131111.c3d1e78-2ubuntu1.1 all
> [installed,automatic]
> nova-compute-qemu/mos9.0-updates 2:13.1.4-7~u14.04+mos38 all
> qemu/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
> qemu-block-extra/mos9.0-updates,now 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
> [installed,automatic]
> qemu-guest-agent/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
> qemu-kvm/mos9.0-updates,now 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64 [installed]
> qemu-slof/trusty-updates 20140630+dfsg-1ubuntu1~14.04 all
> qemu-system/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
> qemu-system-arm/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
> qemu-system-common/mos9.0-updates,now 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
> [installed,automatic]
> qemu-system-mips/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
> qemu-system-misc/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
> qemu-system-ppc/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
> qemu-system-sparc/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
> qemu-system-x86/mos9.0-updates,now 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
> [installed,automatic]
> qemu-user/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
> qemu-user-binfmt/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
> qemu-user-static/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
> qemu-utils/mos9.0-updates,now 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
> [installed,automatic]
>
>
> Qemu command line with multiqueue enabled:
>
> root at blade0-7:~# ps aux|grep qemu
> libvirt+ 18516  100  0.1 64700976 75052 ?      Sl   15:38   1:41
> qemu-system-x86_64 -enable-kvm -name instance-00000018 -S -machine
> pc-i440fx-wily,accel=kvm,usb=off -cpu host -m 59392 -realtime mlock=off -smp
> 16,sockets=8,cores=1,threads=2 -object
> memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/mnt/hugepages_1GB/libvirt/qemu,share=yes,size=62277025792,host-nodes=0,policy=bind
> -numa node,nodeid=0,cpus=0-15,memdev=ram-node0 -uuid
> 40f79f06-fd66-47f1-a952-cb1366117c15 -smbios type=1,manufacturer=OpenStack
> Foundation,product=OpenStack
> Nova,version=13.1.4,serial=db887cbe-039b-453f-9eb9-4443a3ac48e5,uuid=40f79f06-fd66-47f1-a952-cb1366117c15,family=Virtual
> Machine -no-user-config -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-instance-00000018/monitor.sock,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
> -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot
> strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
> file=rbd:compute/40f79f06-fd66-47f1-a952-cb1366117c15_disk:id=compute:key=AQC8Z/hZiuytBRAAS9nrU5BkLQyVnngO/1az2A==:auth_supported=cephx\;none:mon_host=192.168.1.1\:6789,format=raw,if=none,id=drive-virtio-disk0,cache=writeback
> -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> -drive
> file=rbd:compute/40f79f06-fd66-47f1-a952-cb1366117c15_disk.config:id=compute:key=AQC8Z/hZiuytBRAAS9nrU5BkLQyVnngO/1az2A==:auth_supported=cephx\;none:mon_host=192.168.1.1\:6789,format=raw,if=none,id=drive-virtio-disk25,cache=writeback
> -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk25,id=virtio-disk25
> -chardev socket,id=charnet0,path=/var/run/openvswitch/vhu5ff7901d-17 -netdev
> type=vhost-user,id=hostnet0,chardev=charnet0,queues=16 -device
> virtio-net-pci,mq=on,vectors=34,netdev=hostnet0,id=net0,mac=02:10:20:10:03:00,bus=pci.0,addr=0x3
> -chardev socket,id=charnet1,path=/var/run/openvswitch/vhu6137cefb-a5 -netdev
> type=vhost-user,id=hostnet1,chardev=charnet1,queues=16 -device
> virtio-net-pci,mq=on,vectors=34,netdev=hostnet1,id=net1,mac=02:10:20:10:03:02,bus=pci.0,addr=0x4
> -chardev socket,id=charnet2,path=/var/run/openvswitch/vhu98d81e14-60 -netdev
> type=vhost-user,id=hostnet2,chardev=charnet2,queues=16 -device
> virtio-net-pci,mq=on,vectors=34,netdev=hostnet2,id=net2,mac=02:10:20:10:03:04,bus=pci.0,addr=0x5
> -chardev socket,id=charnet3,path=/var/run/openvswitch/vhu38230b89-15 -netdev
> type=vhost-user,id=hostnet3,chardev=charnet3,queues=16 -device
> virtio-net-pci,mq=on,vectors=34,netdev=hostnet3,id=net3,mac=02:10:20:10:03:05,bus=pci.0,addr=0x6
> -chardev socket,id=charnet4,path=/var/run/openvswitch/vhu173ca317-4f -netdev
> type=vhost-user,id=hostnet4,chardev=charnet4,queues=16 -device
> virtio-net-pci,mq=on,vectors=34,netdev=hostnet4,id=net4,mac=02:10:20:10:03:06,bus=pci.0,addr=0x7
> -chardev
> file,id=charserial0,path=/var/lib/nova/instances/40f79f06-fd66-47f1-a952-cb1366117c15/console.log
> -device isa-serial,chardev=charserial0,id=serial0 -chardev
> pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device
> usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -device
> cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
> i6300esb,id=watchdog0,bus=pci.0,addr=0xb -watchdog-action reset -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa -msg timestamp=on
> root at blade0-7:~#
>
>
> Without multiqueue:
>
> root at blade0-7:~# ps aux|grep qemu
> libvirt+ 12808  277  0.3 65044396 201752 ?     Sl   Nov16 17003:49
> qemu-system-x86_64 -enable-kvm -name instance-00000018 -S -machine
> pc-i440fx-wily,accel=kvm,usb=off -cpu host -m 59392 -realtime mlock=off -smp
> 16,sockets=8,cores=1,threads=2 -object
> memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/mnt/hugepages_1GB/libvirt/qemu,share=yes,size=62277025792,host-nodes=0,policy=bind
> -numa node,nodeid=0,cpus=0-15,memdev=ram-node0 -uuid
> 40f79f06-fd66-47f1-a952-cb1366117c15 -smbios type=1,manufacturer=OpenStack
> Foundation,product=OpenStack
> Nova,version=13.1.4,serial=db887cbe-039b-453f-9eb9-4443a3ac48e5,uuid=40f79f06-fd66-47f1-a952-cb1366117c15,family=Virtual
> Machine -no-user-config -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-instance-00000018/monitor.sock,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
> -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot
> strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
> file=rbd:compute/40f79f06-fd66-47f1-a952-cb1366117c15_disk:id=compute:key=AQC8Z/hZiuytBRAAS9nrU5BkLQyVnngO/1az2A==:auth_supported=cephx\;none:mon_host=192.168.1.1\:6789,format=raw,if=none,id=drive-virtio-disk0,cache=writeback
> -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> -drive
> file=rbd:compute/40f79f06-fd66-47f1-a952-cb1366117c15_disk.config:id=compute:key=AQC8Z/hZiuytBRAAS9nrU5BkLQyVnngO/1az2A==:auth_supported=cephx\;none:mon_host=192.168.1.1\:6789,format=raw,if=none,id=drive-virtio-disk25,cache=writeback
> -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk25,id=virtio-disk25
> -chardev socket,id=charnet0,path=/var/run/openvswitch/vhu5ff7901d-17 -netdev
> type=vhost-user,id=hostnet0,chardev=charnet0 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=02:10:20:10:03:00,bus=pci.0,addr=0x3
> -chardev socket,id=charnet1,path=/var/run/openvswitch/vhu6137cefb-a5 -netdev
> type=vhost-user,id=hostnet1,chardev=charnet1 -device
> virtio-net-pci,netdev=hostnet1,id=net1,mac=02:10:20:10:03:02,bus=pci.0,addr=0x4
> -chardev socket,id=charnet2,path=/var/run/openvswitch/vhu98d81e14-60 -netdev
> type=vhost-user,id=hostnet2,chardev=charnet2 -device
> virtio-net-pci,netdev=hostnet2,id=net2,mac=02:10:20:10:03:04,bus=pci.0,addr=0x5
> -chardev socket,id=charnet3,path=/var/run/openvswitch/vhu38230b89-15 -netdev
> type=vhost-user,id=hostnet3,chardev=charnet3 -device
> virtio-net-pci,netdev=hostnet3,id=net3,mac=02:10:20:10:03:05,bus=pci.0,addr=0x6
> -chardev socket,id=charnet4,path=/var/run/openvswitch/vhu173ca317-4f -netdev
> type=vhost-user,id=hostnet4,chardev=charnet4 -device
> virtio-net-pci,netdev=hostnet4,id=net4,mac=02:10:20:10:03:06,bus=pci.0,addr=0x7
> -chardev
> file,id=charserial0,path=/var/lib/nova/instances/40f79f06-fd66-47f1-a952-cb1366117c15/console.log
> -device isa-serial,chardev=charserial0,id=serial0 -chardev
> pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device
> usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -device
> cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
> i6300esb,id=watchdog0,bus=pci.0,addr=0xb -watchdog-action reset -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa -msg timestamp=on
> root at blade0-7:~#
>
> BR,
> Zoltan
>
>
> ________________________________
> From: Ladi Prosek <lprosek at redhat.com>
> Sent: Tuesday, November 14, 2017 7:58 AM
> To: Rafael Gellert
> Cc: ipxe-devel at lists.ipxe.org
> Subject: Re: [ipxe-devel] iPXE boot fails if multique is enabled in
> Openstack
>
> Hi,
>
> On Mon, Nov 13, 2017 at 8:33 AM, Rafael Gellert
> <rafael.gellert at ericsson.com> wrote:
>> Hi,
>>
>> Could you please help me resolve this issue:
>> http://forum.ipxe.org/showthread.php?tid=10521
>
> Can you please post the QEMU command line with and without the
> problematic hw_vif_multiqueue_enabled="true" option. Also QEMU version
> running on the host.
>
> Thank you,
> Ladi
>
>