<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<font size="+1"><font face="FreeSerif">Hi Ladi,<br>
<br>
That's great news, good to see that the problem is reproducible!<br>
Please take your time, we are waiting for the analysis.<br>
<br>
Thanks,<br>
Zoltan<br>
</font></font><br>
<div class="moz-cite-prefix">On 11/24/2017 04:01 PM, Ladi Prosek
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CABdb737zzpu=dsBp-gUTsKdNuTWZO-of3pjVSPaJCdZ3xx=QwA@mail.gmail.com">
<pre wrap="">On Wed, Nov 22, 2017 at 8:10 PM, Ladi Prosek <a class="moz-txt-link-rfc2396E" href="mailto:lprosek@redhat.com"><lprosek@redhat.com></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Indeed, "VIRTIO-NET %p tx complete iobuf %p\n" would be printed if the
TX packets were processed.
I'll try to set up a local repro to investigate.
</pre>
</blockquote>
<pre wrap="">
Quick update: I was able to set up OVS with DPDK and see the same
behavior as you guys, i.e. packets in the (first) tx queue seem to be
ignored by the backend if virtio-net is configured with more than one
queue.
This is not really my forte, please give me more time to drill into it.
</pre>
<blockquote type="cite">
<pre wrap="">Thanks,
Ladi
On Wed, Nov 22, 2017 at 5:06 PM, Zoltan Kanizsai
<a class="moz-txt-link-rfc2396E" href="mailto:zoltan.kanizsai@ericsson.com"><zoltan.kanizsai@ericsson.com></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Ladi,
Thanks for the effort!
The main problem is not with incoming DHCP Replies, but that DHCP Request
packets cannot be sent out from the VM by the iPXE image: TX errors present.
And I can see this also when using ovs-tcpdump on the vhost-user socket:
nothing comes out from the VM trying to pxe boot.
BR,
Zoltan
On 11/22/2017 04:57 PM, Ladi Prosek wrote:
Hi Zoltan,
I'll work on setting this up on my host in the upcoming days. Adding
Victor to cc in case this is a known issue or he knows off-hand why
this doesn't work.
In summary, this is vhost-user networking configured with mq=on but
the simple iPXE driver uses only the first queue. Maybe incoming DHCP
packets are delivered to one of the other queues, which is why the
driver doesn't see them.
Thanks,
Ladi
On Mon, Nov 20, 2017 at 4:44 PM, Zoltan Kanizsai
<a class="moz-txt-link-rfc2396E" href="mailto:zoltan.kanizsai@ericsson.com"><zoltan.kanizsai@ericsson.com></a> wrote:
Hi Ladi,
I'm in the same team with Rafael. Here are the information you asked for:
Qemu related package versions:
root@blade0-7:~# apt list|grep qemu
WARNING: apt does not have a stable CLI interface yet. Use with caution in
scripts.
ipxe-qemu/trusty-updates,now 1.0.0+git-20131111.c3d1e78-2ubuntu1.1 all
[installed,automatic]
nova-compute-qemu/mos9.0-updates 2:13.1.4-7~u14.04+mos38 all
qemu/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
qemu-block-extra/mos9.0-updates,now 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
[installed,automatic]
qemu-guest-agent/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
qemu-kvm/mos9.0-updates,now 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64 [installed]
qemu-slof/trusty-updates 20140630+dfsg-1ubuntu1~14.04 all
qemu-system/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
qemu-system-arm/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
qemu-system-common/mos9.0-updates,now 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
[installed,automatic]
qemu-system-mips/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
qemu-system-misc/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
qemu-system-ppc/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
qemu-system-sparc/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
qemu-system-x86/mos9.0-updates,now 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
[installed,automatic]
qemu-user/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
qemu-user-binfmt/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
qemu-user-static/mos9.0-updates 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
qemu-utils/mos9.0-updates,now 1:2.5+dfsg-5ubuntu6~u1404+mos4 amd64
[installed,automatic]
Qemu command line with multiqueue enabled:
root@blade0-7:~# ps aux|grep qemu
libvirt+ 18516 100 0.1 64700976 75052 ? Sl 15:38 1:41
qemu-system-x86_64 -enable-kvm -name instance-00000018 -S -machine
pc-i440fx-wily,accel=kvm,usb=off -cpu host -m 59392 -realtime mlock=off -smp
16,sockets=8,cores=1,threads=2 -object
memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/mnt/hugepages_1GB/libvirt/qemu,share=yes,size=62277025792,host-nodes=0,policy=bind
-numa node,nodeid=0,cpus=0-15,memdev=ram-node0 -uuid
40f79f06-fd66-47f1-a952-cb1366117c15 -smbios type=1,manufacturer=OpenStack
Foundation,product=OpenStack
Nova,version=13.1.4,serial=db887cbe-039b-453f-9eb9-4443a3ac48e5,uuid=40f79f06-fd66-47f1-a952-cb1366117c15,family=Virtual
Machine -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-instance-00000018/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
-global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot
strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=rbd:compute/40f79f06-fd66-47f1-a952-cb1366117c15_disk:id=compute:key=AQC8Z/hZiuytBRAAS9nrU5BkLQyVnngO/1az2A==:auth_supported=cephx\;none:mon_host=192.168.1.1\:6789,format=raw,if=none,id=drive-virtio-disk0,cache=writeback
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=rbd:compute/40f79f06-fd66-47f1-a952-cb1366117c15_disk.config:id=compute:key=AQC8Z/hZiuytBRAAS9nrU5BkLQyVnngO/1az2A==:auth_supported=cephx\;none:mon_host=192.168.1.1\:6789,format=raw,if=none,id=drive-virtio-disk25,cache=writeback
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk25,id=virtio-disk25
-chardev socket,id=charnet0,path=/var/run/openvswitch/vhu5ff7901d-17 -netdev
type=vhost-user,id=hostnet0,chardev=charnet0,queues=16 -device
virtio-net-pci,mq=on,vectors=34,netdev=hostnet0,id=net0,mac=02:10:20:10:03:00,bus=pci.0,addr=0x3
-chardev socket,id=charnet1,path=/var/run/openvswitch/vhu6137cefb-a5 -netdev
type=vhost-user,id=hostnet1,chardev=charnet1,queues=16 -device
virtio-net-pci,mq=on,vectors=34,netdev=hostnet1,id=net1,mac=02:10:20:10:03:02,bus=pci.0,addr=0x4
-chardev socket,id=charnet2,path=/var/run/openvswitch/vhu98d81e14-60 -netdev
type=vhost-user,id=hostnet2,chardev=charnet2,queues=16 -device
virtio-net-pci,mq=on,vectors=34,netdev=hostnet2,id=net2,mac=02:10:20:10:03:04,bus=pci.0,addr=0x5
-chardev socket,id=charnet3,path=/var/run/openvswitch/vhu38230b89-15 -netdev
type=vhost-user,id=hostnet3,chardev=charnet3,queues=16 -device
virtio-net-pci,mq=on,vectors=34,netdev=hostnet3,id=net3,mac=02:10:20:10:03:05,bus=pci.0,addr=0x6
-chardev socket,id=charnet4,path=/var/run/openvswitch/vhu173ca317-4f -netdev
type=vhost-user,id=hostnet4,chardev=charnet4,queues=16 -device
virtio-net-pci,mq=on,vectors=34,netdev=hostnet4,id=net4,mac=02:10:20:10:03:06,bus=pci.0,addr=0x7
-chardev
file,id=charserial0,path=/var/lib/nova/instances/40f79f06-fd66-47f1-a952-cb1366117c15/console.log
-device isa-serial,chardev=charserial0,id=serial0 -chardev
pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device
usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -device
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
i6300esb,id=watchdog0,bus=pci.0,addr=0xb -watchdog-action reset -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa -msg timestamp=on
root@blade0-7:~#
Without multiqueue:
root@blade0-7:~# ps aux|grep qemu
libvirt+ 12808 277 0.3 65044396 201752 ? Sl Nov16 17003:49
qemu-system-x86_64 -enable-kvm -name instance-00000018 -S -machine
pc-i440fx-wily,accel=kvm,usb=off -cpu host -m 59392 -realtime mlock=off -smp
16,sockets=8,cores=1,threads=2 -object
memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/mnt/hugepages_1GB/libvirt/qemu,share=yes,size=62277025792,host-nodes=0,policy=bind
-numa node,nodeid=0,cpus=0-15,memdev=ram-node0 -uuid
40f79f06-fd66-47f1-a952-cb1366117c15 -smbios type=1,manufacturer=OpenStack
Foundation,product=OpenStack
Nova,version=13.1.4,serial=db887cbe-039b-453f-9eb9-4443a3ac48e5,uuid=40f79f06-fd66-47f1-a952-cb1366117c15,family=Virtual
Machine -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-instance-00000018/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
-global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot
strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=rbd:compute/40f79f06-fd66-47f1-a952-cb1366117c15_disk:id=compute:key=AQC8Z/hZiuytBRAAS9nrU5BkLQyVnngO/1az2A==:auth_supported=cephx\;none:mon_host=192.168.1.1\:6789,format=raw,if=none,id=drive-virtio-disk0,cache=writeback
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=rbd:compute/40f79f06-fd66-47f1-a952-cb1366117c15_disk.config:id=compute:key=AQC8Z/hZiuytBRAAS9nrU5BkLQyVnngO/1az2A==:auth_supported=cephx\;none:mon_host=192.168.1.1\:6789,format=raw,if=none,id=drive-virtio-disk25,cache=writeback
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x9,drive=drive-virtio-disk25,id=virtio-disk25
-chardev socket,id=charnet0,path=/var/run/openvswitch/vhu5ff7901d-17 -netdev
type=vhost-user,id=hostnet0,chardev=charnet0 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=02:10:20:10:03:00,bus=pci.0,addr=0x3
-chardev socket,id=charnet1,path=/var/run/openvswitch/vhu6137cefb-a5 -netdev
type=vhost-user,id=hostnet1,chardev=charnet1 -device
virtio-net-pci,netdev=hostnet1,id=net1,mac=02:10:20:10:03:02,bus=pci.0,addr=0x4
-chardev socket,id=charnet2,path=/var/run/openvswitch/vhu98d81e14-60 -netdev
type=vhost-user,id=hostnet2,chardev=charnet2 -device
virtio-net-pci,netdev=hostnet2,id=net2,mac=02:10:20:10:03:04,bus=pci.0,addr=0x5
-chardev socket,id=charnet3,path=/var/run/openvswitch/vhu38230b89-15 -netdev
type=vhost-user,id=hostnet3,chardev=charnet3 -device
virtio-net-pci,netdev=hostnet3,id=net3,mac=02:10:20:10:03:05,bus=pci.0,addr=0x6
-chardev socket,id=charnet4,path=/var/run/openvswitch/vhu173ca317-4f -netdev
type=vhost-user,id=hostnet4,chardev=charnet4 -device
virtio-net-pci,netdev=hostnet4,id=net4,mac=02:10:20:10:03:06,bus=pci.0,addr=0x7
-chardev
file,id=charserial0,path=/var/lib/nova/instances/40f79f06-fd66-47f1-a952-cb1366117c15/console.log
-device isa-serial,chardev=charserial0,id=serial0 -chardev
pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device
usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -device
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
i6300esb,id=watchdog0,bus=pci.0,addr=0xb -watchdog-action reset -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xa -msg timestamp=on
root@blade0-7:~#
BR,
Zoltan
________________________________
From: Ladi Prosek <a class="moz-txt-link-rfc2396E" href="mailto:lprosek@redhat.com"><lprosek@redhat.com></a>
Sent: Tuesday, November 14, 2017 7:58 AM
To: Rafael Gellert
Cc: <a class="moz-txt-link-abbreviated" href="mailto:ipxe-devel@lists.ipxe.org">ipxe-devel@lists.ipxe.org</a>
Subject: Re: [ipxe-devel] iPXE boot fails if multique is enabled in
Openstack
Hi,
On Mon, Nov 13, 2017 at 8:33 AM, Rafael Gellert
<a class="moz-txt-link-rfc2396E" href="mailto:rafael.gellert@ericsson.com"><rafael.gellert@ericsson.com></a> wrote:
Hi,
Could you please help me resolve this issue:
<a class="moz-txt-link-freetext" href="http://forum.ipxe.org/showthread.php?tid=10521">http://forum.ipxe.org/showthread.php?tid=10521</a>
Can you please post the QEMU command line with and without the
problematic hw_vif_multiqueue_enabled="true" option. Also QEMU version
running on the host.
Thank you,
Ladi
</pre>
</blockquote>
</blockquote>
</blockquote>
<br>
</body>
</html>