[ipxe-devel] iPXE boot fails if multique is enabled in Openstack

Fri Nov 24 17:09:25 UTC 2017

Hi Ladi,

That's great news, good to see that the problem is reproducible!
Please take your time, we are waiting for the analysis.

Thanks,
Zoltan

On 11/24/2017 04:01 PM, Ladi Prosek wrote:
> On Wed, Nov 22, 2017 at 8:10 PM, Ladi Prosek <lprosek at redhat.com> wrote:
>> Indeed, "VIRTIO-NET %p tx complete iobuf %p\n" would be printed if the
>> TX packets were processed.
>>
>> I'll try to set up a local repro to investigate.
> Quick update: I was able to set up OVS with DPDK and see the same
> behavior as you guys, i.e. packets in the (first) tx queue seem to be
> ignored by the backend if virtio-net is configured with more than one
> queue.
>
> This is not really my forte, please give me more time to drill into it.
>
>> Thanks,
>> Ladi
>>
>> On Wed, Nov 22, 2017 at 5:06 PM, Zoltan Kanizsai
>> <zoltan.kanizsai at ericsson.com> wrote:
>>> Hi Ladi,
>>>
>>> Thanks for the effort!
>>>
>>> The main problem is not with incoming DHCP Replies, but that DHCP Request
>>> packets cannot be sent out from the VM by the iPXE image: TX errors present.
>>> And I can see this also when using ovs-tcpdump on the vhost-user socket:
>>> nothing comes out from the VM trying to pxe boot.
>>>
>>> BR,
>>> Zoltan
>>>
>>> On 11/22/2017 04:57 PM, Ladi Prosek wrote:
>>>
>>> Hi Zoltan,
>>>
>>> I'll work on setting this up on my host in the upcoming days. Adding
>>> Victor to cc in case this is a known issue or he knows off-hand why
>>> this doesn't work.
>>>
>>> In summary, this is vhost-user networking configured with mq=on but
>>> the simple iPXE driver uses only the first queue. Maybe incoming DHCP
>>> packets are delivered to one of the other queues, which is why the
>>> driver doesn't see them.
>>>
>>> Thanks,
>>> Ladi
>>>
>>> On Mon, Nov 20, 2017 at 4:44 PM, Zoltan Kanizsai
>>> <zoltan.kanizsai at ericsson.com> wrote:
>>>
>>> Hi Ladi,
>>>
>>> I'm in the same team with Rafael. Here are the information you asked for:
>>>
>>> Qemu related package versions:
>>>
>>> root at blade0-7:~# apt list|grep qemu
>>>
>>> WARNING: apt does not have a stable CLI interface yet. Use with caution in
>>> scripts.
>>>
>>> ipxe-qemu/trusty-updates,now 1.0.0+git-20131111.c3d1e78-2ubuntu1.1 all
>>> [installed,automatic]
>>> nova-compute-qemu/mos9.0-updates 2:13.1.4-7~u14.04+mos38 all
>>> qemu/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
>>> qemu-block-extra/mos9.0-updates,now 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
>>> [installed,automatic]
>>> qemu-guest-agent/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
>>> qemu-kvm/mos9.0-updates,now 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64 [installed]
>>> qemu-slof/trusty-updates 20140630+dfsg-1ubuntu1~14.04 all
>>> qemu-system/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
>>> qemu-system-arm/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
>>> qemu-system-common/mos9.0-updates,now 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
>>> [installed,automatic]
>>> qemu-system-mips/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
>>> qemu-system-misc/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
>>> qemu-system-ppc/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
>>> qemu-system-sparc/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
>>> qemu-system-x86/mos9.0-updates,now 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
>>> [installed,automatic]
>>> qemu-user/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
>>> qemu-user-binfmt/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
>>> qemu-user-static/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
>>> qemu-utils/mos9.0-updates,now 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
>>> [installed,automatic]
>>>
>>>
>>> Qemu command line with multiqueue enabled:
>>>
>>> root at blade0-7:~# ps aux|grep qemu
>>> libvirt+ 18516  100  0.1 64700976 75052 ?      Sl   15:38   1:41
>>> qemu-system-x86_64 -enable-kvm -name instance-00000018 -S -machine
>>> pc-i440fx-wily,accel=kvm,usb=off -cpu host -m 59392 -realtime mlock=off -smp
>>> 16,sockets=8,cores=1,threads=2 -object
>>> memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/mnt/hugepages_1GB/libvirt/qemu,share=yes,size=62277025792,host-nodes=0,policy=bind
>>> -numa node,nodeid=0,cpus=0-15,memdev=ram-node0 -uuid
>>> 40f79f06-fd66-47f1-a952-cb1366117c15 -smbios type=1,manufacturer=OpenStack
>>> Foundation,product=OpenStack
>>> Nova,version=13.1.4,serial=db887cbe-039b-453f-9eb9-4443a3ac48e5,uuid=40f79f06-fd66-47f1-a952-cb1366117c15,family=Virtual
>>> Machine -no-user-config -nodefaults -chardev
>>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-instance-00000018/monitor.sock,server,nowait
>>> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
>>> -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot
>>> strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
>>> file=rbd:compute/40f79f06-fd66-47f1-a952-cb1366117c15_disk:id=compute:key=AQC8Z/hZiuytBRAAS9nrU5BkLQyVnngO/1az2A==:auth_supported=cephx\;none:mon_host=192.168.1.1\:6789,format=raw,if=none,id=drive-virtio-disk0,cache=writeback
>>> -device
>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>>> -drive
>>> file=rbd:compute/40f79f06-fd66-47f1-a952-cb1366117c15_disk.config:id=compute:key=AQC8Z/hZiuytBRAAS9nrU5BkLQyVnngO/1az2A==:auth_supported=cephx\;none:mon_host=192.168.1.1\:6789,format=raw,if=none,id=drive-virtio-disk25,cache=writeback
>>> -device
>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk25,id=virtio-disk25
>>> -chardev socket,id=charnet0,path=/var/run/openvswitch/vhu5ff7901d-17 -netdev
>>> type=vhost-user,id=hostnet0,chardev=charnet0,queues=16 -device
>>> virtio-net-pci,mq=on,vectors=34,netdev=hostnet0,id=net0,mac=02:10:20:10:03:00,bus=pci.0,addr=0x3
>>> -chardev socket,id=charnet1,path=/var/run/openvswitch/vhu6137cefb-a5 -netdev
>>> type=vhost-user,id=hostnet1,chardev=charnet1,queues=16 -device
>>> virtio-net-pci,mq=on,vectors=34,netdev=hostnet1,id=net1,mac=02:10:20:10:03:02,bus=pci.0,addr=0x4
>>> -chardev socket,id=charnet2,path=/var/run/openvswitch/vhu98d81e14-60 -netdev
>>> type=vhost-user,id=hostnet2,chardev=charnet2,queues=16 -device
>>> virtio-net-pci,mq=on,vectors=34,netdev=hostnet2,id=net2,mac=02:10:20:10:03:04,bus=pci.0,addr=0x5
>>> -chardev socket,id=charnet3,path=/var/run/openvswitch/vhu38230b89-15 -netdev
>>> type=vhost-user,id=hostnet3,chardev=charnet3,queues=16 -device
>>> virtio-net-pci,mq=on,vectors=34,netdev=hostnet3,id=net3,mac=02:10:20:10:03:05,bus=pci.0,addr=0x6
>>> -chardev socket,id=charnet4,path=/var/run/openvswitch/vhu173ca317-4f -netdev
>>> type=vhost-user,id=hostnet4,chardev=charnet4,queues=16 -device
>>> virtio-net-pci,mq=on,vectors=34,netdev=hostnet4,id=net4,mac=02:10:20:10:03:06,bus=pci.0,addr=0x7
>>> -chardev
>>> file,id=charserial0,path=/var/lib/nova/instances/40f79f06-fd66-47f1-a952-cb1366117c15/console.log
>>> -device isa-serial,chardev=charserial0,id=serial0 -chardev
>>> pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device
>>> usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -device
>>> cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
>>> i6300esb,id=watchdog0,bus=pci.0,addr=0xb -watchdog-action reset -device
>>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa -msg timestamp=on
>>> root at blade0-7:~#
>>>
>>>
>>> Without multiqueue:
>>>
>>> root at blade0-7:~# ps aux|grep qemu
>>> libvirt+ 12808  277  0.3 65044396 201752 ?     Sl   Nov16 17003:49
>>> qemu-system-x86_64 -enable-kvm -name instance-00000018 -S -machine
>>> pc-i440fx-wily,accel=kvm,usb=off -cpu host -m 59392 -realtime mlock=off -smp
>>> 16,sockets=8,cores=1,threads=2 -object
>>> memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/mnt/hugepages_1GB/libvirt/qemu,share=yes,size=62277025792,host-nodes=0,policy=bind
>>> -numa node,nodeid=0,cpus=0-15,memdev=ram-node0 -uuid
>>> 40f79f06-fd66-47f1-a952-cb1366117c15 -smbios type=1,manufacturer=OpenStack
>>> Foundation,product=OpenStack
>>> Nova,version=13.1.4,serial=db887cbe-039b-453f-9eb9-4443a3ac48e5,uuid=40f79f06-fd66-47f1-a952-cb1366117c15,family=Virtual
>>> Machine -no-user-config -nodefaults -chardev
>>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-instance-00000018/monitor.sock,server,nowait
>>> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
>>> -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot
>>> strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
>>> file=rbd:compute/40f79f06-fd66-47f1-a952-cb1366117c15_disk:id=compute:key=AQC8Z/hZiuytBRAAS9nrU5BkLQyVnngO/1az2A==:auth_supported=cephx\;none:mon_host=192.168.1.1\:6789,format=raw,if=none,id=drive-virtio-disk0,cache=writeback
>>> -device
>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>>> -drive
>>> file=rbd:compute/40f79f06-fd66-47f1-a952-cb1366117c15_disk.config:id=compute:key=AQC8Z/hZiuytBRAAS9nrU5BkLQyVnngO/1az2A==:auth_supported=cephx\;none:mon_host=192.168.1.1\:6789,format=raw,if=none,id=drive-virtio-disk25,cache=writeback
>>> -device
>>> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk25,id=virtio-disk25
>>> -chardev socket,id=charnet0,path=/var/run/openvswitch/vhu5ff7901d-17 -netdev
>>> type=vhost-user,id=hostnet0,chardev=charnet0 -device
>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=02:10:20:10:03:00,bus=pci.0,addr=0x3
>>> -chardev socket,id=charnet1,path=/var/run/openvswitch/vhu6137cefb-a5 -netdev
>>> type=vhost-user,id=hostnet1,chardev=charnet1 -device
>>> virtio-net-pci,netdev=hostnet1,id=net1,mac=02:10:20:10:03:02,bus=pci.0,addr=0x4
>>> -chardev socket,id=charnet2,path=/var/run/openvswitch/vhu98d81e14-60 -netdev
>>> type=vhost-user,id=hostnet2,chardev=charnet2 -device
>>> virtio-net-pci,netdev=hostnet2,id=net2,mac=02:10:20:10:03:04,bus=pci.0,addr=0x5
>>> -chardev socket,id=charnet3,path=/var/run/openvswitch/vhu38230b89-15 -netdev
>>> type=vhost-user,id=hostnet3,chardev=charnet3 -device
>>> virtio-net-pci,netdev=hostnet3,id=net3,mac=02:10:20:10:03:05,bus=pci.0,addr=0x6
>>> -chardev socket,id=charnet4,path=/var/run/openvswitch/vhu173ca317-4f -netdev
>>> type=vhost-user,id=hostnet4,chardev=charnet4 -device
>>> virtio-net-pci,netdev=hostnet4,id=net4,mac=02:10:20:10:03:06,bus=pci.0,addr=0x7
>>> -chardev
>>> file,id=charserial0,path=/var/lib/nova/instances/40f79f06-fd66-47f1-a952-cb1366117c15/console.log
>>> -device isa-serial,chardev=charserial0,id=serial0 -chardev
>>> pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device
>>> usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -device
>>> cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
>>> i6300esb,id=watchdog0,bus=pci.0,addr=0xb -watchdog-action reset -device
>>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa -msg timestamp=on
>>> root at blade0-7:~#
>>>
>>> BR,
>>> Zoltan
>>>
>>>
>>> ________________________________
>>> From: Ladi Prosek <lprosek at redhat.com>
>>> Sent: Tuesday, November 14, 2017 7:58 AM
>>> To: Rafael Gellert
>>> Cc: ipxe-devel at lists.ipxe.org
>>> Subject: Re: [ipxe-devel] iPXE boot fails if multique is enabled in
>>> Openstack
>>>
>>> Hi,
>>>
>>> On Mon, Nov 13, 2017 at 8:33 AM, Rafael Gellert
>>> <rafael.gellert at ericsson.com> wrote:
>>>
>>> Hi,
>>>
>>> Could you please help me resolve this issue:
>>> http://forum.ipxe.org/showthread.php?tid=10521
>>>
>>> Can you please post the QEMU command line with and without the
>>> problematic hw_vif_multiqueue_enabled="true" option. Also QEMU version
>>> running on the host.
>>>
>>> Thank you,
>>> Ladi
>>>
>>>
>>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ipxe.org/pipermail/ipxe-devel/attachments/20171124/7ef6b1e6/attachment.htm>